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Abstract 

Assessing  the  effectiveness  of  humanitarian  assistance  and  foreign  aid  has 
presented  a  challenge  to  personnel  tasked  with  these  operations.  Answering  the  question 
“are  we  winning  hearts  and  minds”  has  similarly  eluded  military  personnel  in  Iraq  and 
Afghanistan.  This  research  presents  a  review  of  humanitarian  assistance  and  foreign  aid 
conducted  by  the  U.S.  government  and  literature  on  U.S.  military  doctrine  regarding 
humanitarian  assistance  and  infrastructure  investment.  The  research  builds  upon 
expectancy  disconfirmation  theory  to  determine  the  strongest  predictors  of  citizen 
satisfaction  with  government  services.  Case  study  data  was  collected  in  Belize  before 
and  after  a  U.S.  military  humanitarian  and  civil  assistance  construction  project  was 
executed,  and  this  data  was  analyzed  using  an  expectancy  theory  model.  The  results 
indicate  that  the  model  using  performance,  disconfirmation,  and  an  interaction  effect  of 
both  explains  56%  of  the  variation  in  citizen  satisfaction  and  proposes  a  predictive  model 
of  citizen  satisfaction.  This  article  proposes  further  research  with  improvement  to  the 
survey  methods  and  instrument;  it  also  discusses  how  the  model  may  not  account  for  an 
unmeasured  variable.  Further  research  is  also  suggested  to  determine  the  relationship  of 
time  and  location  on  a  citizen’s  satisfaction  rating  when  considering  the  impact  of  a 
humanitarian  project. 
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PREDICTING  CITIZEN  SATISFACTION  WITH  GOVERNMENT  SERVICES  IN 

BELIZE 


I.  Introduction 

The  United  States  (U.S.)  military  has  been  heavily  involved  in  combat  operations 
since  the  invasions  of  Afghanistan  and  Iraq  in  2001  and  2003,  respectively.  In  both  of 
these  conflicts,  the  U.S.  military  has  executed  counterinsurgency  (COIN)  campaigns  with 
a  heavy  emphasis  on  humanitarian  assistance  projects.  Enormous  sums  of  money  have 
been  spent  in  these  countries,  but  measuring  the  effectiveness  of  these  projects  has  been 
problematic.  While  many  measures  of  performance  have  been  used,  such  as  the  number 
of  projects  completed  and  dollars  spent,  measures  of  effectiveness  of  a  given  project  are 
rarely  implemented.  The  recent  use  of  COIN  strategies  by  the  U.S.  military  and  the 
increased  emphasis  on  operations  intended  to  build  partnerships  with  developing  nations 
necessitates  further  development  of  measures  of  effectiveness  for  these  non-combat 
operations.  The  main  effort  of  this  research  is  to  evaluate  a  survey  method  for  measuring 
the  effectiveness  of  humanitarian  assistance  projects. 

This  thesis  is  in  the  scholarly  article  format.  Chapter  I  begins  by  presenting 
background  information  on  U.S.  government  humanitarian  assistance  programs.  Next, 
the  chapter  presents  the  research  questions,  methodology,  and  scope  of  the  research.  The 
chapter  then  presents  more  in-depth  background  on  U.S  government  humanitarian 
assistance  programs,  U.S.  military  doctrine,  the  impact  of  humanitarian  assistance,  and 
models  for  explaining  citizen  satisfaction.  The  chapter  proceeds  with  background  on 
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Belize,  the  location  of  the  case  study,  before  detailing  aspects  of  Operation  NEW 
HORIZONS  14.  The  chapter  then  discusses  the  experiment  design  and  survey 
procedures  before  concluding  with  a  discussion  of  the  anticipated  significance  of  the 
research.  Chapter  II  is  an  article  to  be  submitted  for  publication  with  the  results  of  this 
research,  and  Chapter  III  reviews  the  research  questions,  provides  more  discussion 
regarding  the  results,  and  concludes  with  recommendations  for  further  research. 

Brief  Background 

The  U.S.  government  executes  humanitarian  assistance  programs  around  the 
world  through  a  myriad  of  programs.  While  these  programs  and  intentions  are  diverse,  a 
common  critique  of  these  programs  is  that  projects  are  poorly  evaluated  for  effectiveness. 
Criticisms  of  these  programs  identify  poorly  defined  goals,  nonexistent  evaluations  of 
past  programs,  and  questionably  quantified  assessments  after  completion  of  projects. 
These  criticisms  have  been  amplified  in  the  wake  of  the  U.S.  invasions  of  Afghanistan 
and  Iraq  after  2001,  where  humanitarian  assistance  programs  have  been  pivotal  elements 
of  the  U.S.  military  COIN  campaigns.  The  U.S.  military  has  outspent  the  post-World 
War  II  European  reconstruction  effort,  the  Marshal  Plan,  on  reconstruction  programs  in 
Afghanistan  since  2001  when  adjusted  for  inflation  to  current  year  dollars  (SIGAR, 

2014). 

Despite  these  massive  humanitarian  activities,  the  question  of  “are  we  winning 
hearts  and  minds”  remains  a  difficult  question  to  answer.  The  difficulty  in  answering  this 
question  is  a  factor  of  the  difficulty  of  measuring  the  effectiveness  of  these  projects. 
Measures  of  effectiveness  (MOE)  are  often  selected  based  on  convenience  or  the  ease  of 
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quantification,  such  as  numbers  of  projects  or  dollars  spent,  instead  of  more  applicable 
metrics  such  as  change  in  perception  of  legitimacy  of  the  host  nation  government  by  their 
population.  Recent  research  also  outlines  examples  where  well-meaning  assistance  may 
be  detrimental  to  security  of  developing  nations  and  stability  of  fragile  governments. 

Problem  Statement  and  Research  Questions 

Determining  the  effectiveness  of  humanitarian  assistance  has  been  a  difficult 
problem.  This  research  will  test  a  method  for  measuring  citizen  satisfaction  with 
government  services  in  Belize  using  citizen  satisfaction  models  developed  regarding 
government  services  in  U.S.  cities.  The  key  research  questions  to  be  answered  in  this 
research  are: 

•  Can  satisfaction  be  predicted  when  the  expectations  of  the  population  is 
known? 

•  Can  satisfaction  be  predicted  when  the  population’s  rating  of  government 
service  performance  is  known? 

•  Can  satisfaction  be  predicted  when  the  difference  between  the  population’s 
expectations  and  perception  of  government  services  performance  is  known? 

•  Which  factors  are  most  influential  on  citizen  satisfaction? 

•  Can  guidance  be  given  in  target  selection  to  increase  return  on  investment  on 
infrastructure  targets  with  respect  to  citizen  satisfaction? 

•  Is  the  model  of  predicting  citizen  satisfaction  sensitive  to  sample  size? 

Methodology 

This  research  employed  citizen  satisfaction  surveys  in  order  to  answer  the 
outlined  research  questions.  Surveys  were  administered  in  conjunction  with  a  U.S. 
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military-led  humanitarian  assistance  mission  in  Belize,  where  samples  were  collected  in 


the  vicinity  of  these  projects.  These  survey  data  were  then  analyzed  to  determine  the 
predictors  of  citizen  satisfaction  using  models  adapted  from  citizen  satisfaction  models 
outlined  in  detail  in  Chapter  I.  Various  regression  techniques  and  dominance  analysis 
were  conducted  on  the  data  to  determine  the  relationships  of  the  predictor  variables  in 
order  to  build  a  predictive  regression  equation.  Further  detail  of  the  survey  instrument 
development,  survey  sampling  plan,  and  experiment  designed  are  also  included  in  this 
chapter,  and  details  of  the  methodologies  used  for  analysis  of  the  results  are  included  in 
Chapter  II. 

Scope  and  Limitations 

This  research  effort  was  conducted  with  another  student  in  the  AFIT  program  and 
was  sponsored  by  Twelfth  Air  Force  (Air  Forces  Southern).  Both  students  used  the  same 
data  collected  through  the  same  survey  method,  but  two  sets  of  analyses  were  conducted. 
This  thesis  research  effort  builds  upon  Hansen’s  (2015)  analysis  and  further  explored  the 
application  of  his  analysis  towards  predicting  citizen  satisfaction.  In  Hansen’s  research, 
the  Van  Ryzin  expectancy-disconfirmation  model  was  adapted  into  the  model  presented 
in  Figure  1,  with  the  interaction  of  expectations  on  disconfirmation  removed  (Hansen, 
2015).  This  research  intends  to  explore  the  combined  effects  of  these  variables  on  citizen 
satisfaction  through  additional  analysis  and  attempts  to  predict  citizen  satisfaction  given 
expectations,  performance,  and  disconfirmation  are  known  within  a  given  population. 
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Figure  1:  Hansen  Research  Model  (Hansen,  2015) 

The  research  efforts  of  the  author  and  Hansen  needed  to  be  separate  and  distinct 
in  order  to  meet  the  thesis  requirement  for  graduation.  While  both  students  worked 
closely  together  throughout  the  research,  each  student  analyzed  different  aspects  of  the 
citizen  satisfaction  research  question  using  different  methodologies.  These  research 
efforts  were  delineated  into  separate  topics  as  illustrated  in  Figure  2.  Hansen  analyzed 
precursors  to  investment  using  Geospatial  Information  Systems  (GIS)  and  tested 
relationships  between  predictor  variables  in  the  citizen  satisfaction  model,  while  this 
research  effort  analyzed  the  citizen  satisfaction  problem  with  a  focus  on  predicting 
satisfaction  and  determining  the  most  significant  predictors  in  the  model. 
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Figure  2:  Delineation  of  Research  Efforts 


Humanitarian  Assistance  Background 

The  United  States  has  been  heavily  involved  in  both  humanitarian  assistance  and 
combat  operations  from  the  second  half  of  the  20th  century  through  present  day. 
American  involvement  in  foreign  aid  began  with  the  passage  of  the  European  Recovery 
Program,  more  commonly  known  as  the  Marshall  Plan,  in  1947  after  the  end  of  World 
War  II.  Between  1948  and  1952,  the  United  States  gave  $13  billion  to  16  European 
countries  for  reconstruction,  which  equates  to  approximately  $103.4  billion  in  current 
year  dollars  (SIGAR,  2014).  From  1952  to  1961,  the  United  States  continued  providing 
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assistance  around  the  world,  and  in  1961  the  United  States  Agency  for  International 
Development  (USAID)  was  established  with  the  passage  of  the  Foreign  Assistance  Act. 
While  USAID  is  the  primary  agency  within  the  federal  government  responsible  for 
foreign  assistance,  the  U.S.  military  is  also  permitted  to  execute  various  humanitarian 
assistance  programs.  Additionally,  many  other  elements  of  the  executive  branch  have 
been  involved  in  humanitarian  assistance  operations  in  Iraq  and  Afghanistan  since  2003. 
A  brief  description  of  each  of  these  major  civilian  and  military  humanitarian  assistance 
programs  is  provided  in  further  detail  in  this  section. 

Civilian  Agency  Foreign  Assistance 

The  Department  of  State  (DoS),  through  the  Director  of  Foreign  Assistance 
(DFA),  is  the  lead  coordinator  within  the  federal  government  for  foreign  assistance. 
USAID  is  the  lead  agency  within  the  federal  government  responsible  for  executing 
economic  development,  health,  governance,  and  disaster  relief  programs,  while  DoS  is 
the  lead  agency  for  executing  law  enforcement,  counter-terrorism,  democracy  promotion, 
refugee  relief,  counter-proliferation,  and  UN-led  peacekeeping  operations  (CRS,  2011). 
The  DFA  has  categorized  all  foreign  assistance  programs  funded  by  DoS  and  USAID 
budgets  into  five  major  program  areas:  Peace  and  Security,  Investing  in  People, 
Governing  Justly  and  Democratically,  Promoting  Economic  Growth  and  Prosperity,  and 
Humanitarian  Assistance.  These  programs  are  funded  through  three  main  budget 
accounts:  Assistance  Serving  Development  and  Humanitarian  Purposes,  Assistance 
Serving  Both  Development  and  Special  Political/Strategic  Purposes,  and  Assistance 
Serving  Security  Purposes  (CRS,  2011).  While  these  accounts  help  organize  programs 
and  funding,  the  current  structure  makes  it  difficult  to  determine  the  exact  amounts  of 
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money  spent  on  humanitarian  assistance  by  civilian  functions  of  the  federal  government. 
All  military  humanitarian  assistance  considered  in  this  research  is  funded  through 
Department  of  Defense  (DoD)  accounts,  which  is  not  tracked  through  DFA  accounts.  In 
total,  these  accounts  were  funded  at  $39.4  billion  in  Fiscal  Year  (FY)  2010,  which 
included  $1.9B  in  Food  Aid  that  is  not  considered  in  the  previous  three  accounts  (CRS, 
2011). 

Department  of  Defense  Foreign  Assistance 

Overseas  Humanitarian,  Disaster,  and  Civic  Aid  (OHDACA)  is  an  appropriation 
for  DoD  that  funds  Humanitarian  Assistance  (HA),  Foreign  Disaster  Response  (FDR), 
and  transportation  of  donated  humanitarian  supplies  to  a  foreign  location.  HA  operations 
are  diverse  and  varied,  ranging  from  supporting  internally  displaced  personnel  or 
refugees,  to  providing  security  for  storage  and  distribution  of  relief  materials,  providing 
technical  assistance  such  as  repairing  communications  infrastructure,  and  training 
personnel  in  procedures  for  demining  operations.  FDR  missions  are  designated  to 
“alleviate  the  suffering  of  foreign  disaster  victims,  including  victims  of  natural  disasters 
and  conflicts,  internally  displaced  persons  (IDPs),  refugees,  stateless  persons,  and 
vulnerable  migrants,”  and  are  authorized  operations  only  if  military  capabilities  are 
unique  or  local  civilian  capability  to  respond  has  been  overwhelmed  (JP  3-29,  2014). 
DoD  has  spent  $328.4  million  between  fiscal  years  2005  and  2010  on  these  operations  in 
every  geographic  combatant  command’s  area  of  responsibility  (AOR),  with  the  highest 
amount  spent  in  U.S.  Southern  Command  (GAO,  2012).  A  graph  of  these  total  amounts 
from  2005  through  2010  is  shown  in  Figure  3  (GAO,  2012). 
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Humanitarian  and  Civic  Assistance  (HCA)  is  a  program  specifically  authorized  by 
Title  10,  USC,  Section  401,  which  authorizes  DoD  to  execute  humanitarian  projects  as 
deployment  training  in  a  foreign  country.  These  activities  are  limited  to  medical 
assistance  in  areas  where  services  are  not  available,  construction  of  surface  transportation 
systems,  construction  of  basic  sanitation  facilities,  and  construction  or  repair  of  public 
facilities  (JP  3-29,  2014).  DoD  has  been  heavily  engaged  in  these  operations,  spending 
$75.1  million  from  fiscal  years  2005  through  2010  in  all  geographic  combatant 
commands’  AORs  except  US  Northern  Command,  with  the  highest  amount  spent  within 
U.S.  Southern  Command  (GAO,  2012).  An  example  of  an  HCA  program  is  Operation 
NEW  HORIZONS  2014,  which  occurred  from  April  to  June  2014  in  Belize  and  is  the 
source  of  survey  data  for  this  research.  A  graph  of  these  total  amounts  from  2005 
through  2010  is  included  in  Figure  3  (GAO,  2012). 

While  the  US  military  has  been  heavily  involved  in  spending  money  and 
executing  projects  through  the  OHDACA  and  HCA  programs,  the  General  Accounting 
Office  (GAO)  has  been  critical  of  its  project  evaluation  process.  DoD  instructions 
mandate  an  after-action  review  within  30  days  of  completion  of  each  OHDACA  project, 
and  all  projects  valued  at  over  $10  thousand  require  another  evaluation  one  year  after 
completion;  however,  the  GAO  found  that  53%  of  completed  projects  did  not  have  an 
after-action  review,  and  90%  of  projects  over  $10  thousand  did  not  have  a  one-year 
evaluation  (GAO,  2012).  While  some  of  the  lack  of  project  evaluations  can  be  attributed 
to  funding  and  logistical  challenges  tied  to  traveling  to  these  locations,  DoD  also 
indicated  that  it  is  difficult  to  assess  the  impact  of  humanitarian  assistance  projects 
(GAO,  2012).  The  GAO  identified  resources  from  RAND  and  the  Sphere  Project  as 
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starting  points  for  the  development  of  these  metrics,  and  cited  USAID  performance 
measurement  planning  as  a  method  for  assessment  (GAO,  2012). 


100 


Figure  3:  OHDACA  and  HCA  Funding  from  2005-2010  (Adapted  from  GAO,  2012) 


Commander’s  Emergency  Response  Program  (CERP) 

The  origins  of  CERP  can  be  traced  to  the  aftermath  of  the  2003  invasion  of  Iraq 
when  US  Forces  recovered  Ba’ath  Party  funds  and  executed  emergency  projects  with  this 
cash.  Within  two  months  of  the  establishment  of  the  Coalition  Provisional  Authority 
(CPA)  in  April  of  2003,  Ambassador  Paul  Bremer  authorized  coalition  forces  to  use  these 
funds  through  the  newly-named  CERP  for  commanders  to  “respond  to  urgent 
humanitarian  relief  and  reconstruction  requirements  within  their  areas  of  responsibility, 
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by  carrying  out  programs  that  will  immediately  assist  the  Iraqi  people  and  support  the 
reconstruction  of  Iraq.”  Further  guidance  from  coalition  commanders  were  promulgated 
in  fragmentary  orders  within  a  few  days,  which  allowed  commanders  to  execute  projects 
including  “the  building,  repair,  reconstitution,  and  reestablishment  of  the  social  and 
material  infrastructure  in  Iraq”  (Martins,  2005).  Initial  limitations  on  these  projects 
restricted  project  approval  levels  to  $100  thousand  for  colonels  and  $500  thousand  for 
major  generals;  furthermore,  coalition  forces  were  prohibited  from  using  these  funds  on 
projects  that  benefited  coalition  forces,  to  purchase  weapons  or  ammunition,  or  pay 
salaries  of  government  employees.  Despite  these  restrictions,  the  CERP  account 
exhausted  Ba’ath  Party  funding  quickly  and  required  $180  million  in  Operations  and 
Maintenance  (O&M)  funding  from  the  $87  billion  emergency  supplemental  funding  bill 
signed  into  law  on  November  6,  2003  (Martins,  2005).  At  this  point,  CERP  had 
transitioned  from  an  Iraqi-funded  to  federally-funded  program.  By  the  time  that  U.S. 
forces  had  left  Iraq  in  December  2011,  U.S.  forces  had  executed  over  36,000  CERP 
projects  at  a  cost  of  $3.7  billion  (SIGIR,  2013a). 

Numerous  reports  by  SIGIR  have  been  critical  about  project  and  contract 
management  of  CERP  projects  in  Iraq.  The  governing  guidance  for  the  use  of  CERP 
funds  was  titled  “Money  as  a  Weapon  System”  (MAAWS),  which  mandated  project 
milestone  and  financial  status  tracking  for  projects.  MAAWS  also  mandated 
coordination  with  USAID  and  Government  of  Iraq  officials  in  response  to  criticism  that 
DoD  was  not  defining  project  requirements  frequently  enough  with  all  stakeholders.  The 
initial  intent  of  CERP  was  to  fund  small,  swift  projects  for  urgent  humanitarian  needs; 
through  2010,  there  were  over  16,000  CERP  projects  each  under  $25  thousand  and  744 
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projects  each  costing  in  excess  of  $500  thousand  dollars  (SIGIR,2013a).  Some  large 
projects  were  attempted  in  which  the  timeline  for  the  project  surpassed  the  project 
manager’s  departure  date;  for  example,  46  projects  worth  $35.5  million  were  executed  at 
Baghdad  International  Airport  over  the  course  of  4  years  by  Civil  Affairs  personnel 
deployed  for  6  to  9  months  at  a  time,  which  resulted  in  only  22  projects  evaluated  as 
successful  (SIGIR,  2013a).  While  program  management  was  a  challenge,  it  was  also 
cited  in  reports  that  goals  and  outcomes  were  poorly  defined  for  many  projects.  Goals 
were  arbitrarily  selected  and  metrics  to  judge  progress  or  success  were  often  absent 
(SIGIR,  2013a). 

CERP  was  authorized  in  both  Afghanistan  and  Iraq  starting  in  FY  2004.  In  both 
theaters,  the  intent  was  to  address  urgent  humanitarian  relief  requirements  through  small, 
quick  projects  in  specific  categories,  with  prohibitions  on  using  the  funds  for  the  benefit 
of  coalition  forces  (SIGAR,  2009).  Unlike  in  Iraq,  CERP  in  Afghanistan  was  always  an 
appropriation  of  federal  funding.  Similar  to  the  experience  in  Iraq,  reports  are  critical 
about  the  management  of  projects.  US  forces  have  spent  $3.7  billion  through  CERP  in 
Afghanistan  from  2004  through  July  2014  (SIGAR,  2014). 

Other  U.S.  Government  Humanitarian  Assistance  Programs  in  Iraq 

In  addition  to  CERP,  the  US  government  spent  $25.7  billion  in  humanitarian 
assistance  from  2003  through  the  end  of  2011.  Of  this  total,  $20.9  billion  was  spent 
through  the  Iraq  Relief  and  Reconstruction  Fund,  and  $4.8  billion  was  spent  through  the 
Economic  Support  Fund  (SIGIR,  2013a).  Iraq  Relief  and  Reconstruction  Fund  projects 
were  primarily  executed  by  the  Department  of  Defense  and  consisted  of  two  separate 
programs,  funded  at  $2.5  billion  and  $18.4  billion,  which  focused  on  infrastructure 
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projects  in  12  critical  sectors  including  electricity,  oil,  healthcare,  transportation, 
education,  and  security.  The  Economic  Support  Fund  was  funded  by  the  Departments  of 
State,  Agriculture,  Justice,  Commerce,  the  Treasury,  as  well  as  USAID,  with  the  focus  on 
economic  development,  job  creation,  community  security,  democracy  promotion,  and 
transitioning  the  Iraqi  economy  away  from  state-owned  enterprises  (SIGIR,  2013b). 
These  totals  do  not  include  funds  spent  on  Iraqi  Security  Forces  or  on  law  enforcement 
operations,  which  are  counted  as  reconstruction  efforts,  but  not  directly  associated  with 
humanitarian  assistance. 

Other  U.S.  Government  Humanitarian  Assistance  Programs  in  Afghanistan 
Excluding  CERP,  there  are  three  major  humanitarian  assistance  funds  in 
Afghanistan,  with  a  total  of  $19.5  billion  spent  from  2002  to  2014.  Of  this  total,  $17.5 
billion  was  spent  on  political,  economic,  and  security  priorities  through  the  USAID- 
funded  Economic  Support  Fund.  Additionally,  the  Department  of  Defense  spent  $1.2 
billion  on  large-scale  public  infrastructure  projects  through  the  Afghanistan  Infrastructure 
Fund;  it  spent  another  $0.8  billion  on  financial  sector  and  banking  reforms  through  the 
Task  Force  for  Business  and  Stability  Operations.  In  total,  these  programs  dwarf  the  $3.7 
billion  in  CERP  funding  over  the  same  time  period  (SIGAR,  2014).  These  totals  do  not 
include  funds  spent  on  counter-narcotics  operations,  law  enforcement  operations,  or  the 
Afghan  Security  Forces.  These  activities  are  considered  reconstruction  efforts,  but 
should  be  considered  separate  from  humanitarian  assistance. 
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Doctrine  Background 

US  military  forces  can  be  employed  throughout  a  wide  spectrum  of  operations  at 
any  location  in  the  world.  These  operations  range  from  peacetime  operations  in  a 
friendly  country,  to  stability  operations  in  a  troubled  nation,  and  ultimately  in  combat 
operations  in  enemy  territory.  The  continuum  of  military  operations  as  accepted  by  US 
military  joint  doctrine  is  illustrated  in  Figure  4  (JP  3-0,  201 1).  While  US  military  forces 
are  trained,  organized,  and  equipped  primarily  for  combat  operations,  these  forces  could 
be  called  upon  to  execute  operations  anywhere  within  this  continuum. 
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Figure  4:  The  Range  of  Military  Operations  (JP  3-0,  2011) 


Humanitarian  assistance  operations  can  be  expected  to  be  executed  throughout  the 
range  of  military  operations.  The  type  and  scale  of  humanitarian  operations  can  be 
expected  to  differ  based  on  the  amount  of  conflict  in  the  Area  of  Responsibility  (AOR) 
of  the  operation;  for  example,  crisis  response  operations  may  have  a  large  humanitarian 
assistance  component,  but  depending  on  the  situation  in  the  objective  country  there  could 
be  no  combat  operations.  These  operations  are  categorized  on  the  range  illustrated  in 
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Figure  4,  with  elements  of  humanitarian  assistance  possible  at  each  level  dependent  on 
where  the  operation  lies  on  the  conflict  continuum  (JP  3-0,  2011).  This  section  will  first 
discuss  the  overall  Joint  Operations  Process,  with  further  discussions  of  peace  operations, 
stability  operations,  and  COIN  operations  as  elements  of  doctrine  at  different  locations 
along  the  conflict  continuum.  The  section  will  conclude  with  a  discussion  on  operations 
assessments  and  their  use  in  humanitarian  assistance. 

The  Joint  Operations  Process 

The  joint  operations  process  is  a  framework  that  outlines  how  the  U.S.  military 
executes  across  the  range  of  military  operations.  These  operations  could  be  anything 
from  peacetime  humanitarian  assistance  projects  to  a  major  theater  war.  For  many  of 
these  various  operations,  there  will  be  an  element  of  civil-military  operations  where 
military  personnel  interface  with  governmental  organizations  and  the  civilian  population 
in  order  to  achieve  military  objectives  and  maximize  civilian  support  for  operations  (JP 
3-0,  201 1).  These  civil-military  operations  may  occur  in  any  phase  during  the  operations 
process  shown  in  Figure  5  (JP  3-0,  2011);  in  particular,  civil-military  operations 
involving  humanitarian  assistance  projects  should  be  expected  to  be  heavy  during  Phases 
0, 1,  IV,  and  V. 
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Figure  5:  Operations  Phases  and  Military  Effort  (JP  3-0,  2011) 


Stability  Operations 

The  U.S.  military  has  been  heavily  involved  in  stability  operations  in  Iraq  and 
Afghanistan  since  2001,  which  consist  of  operations  outside  of  the  U.S.  to  “maintain  or 
reestablish  a  safe  and  secure  environment,  provide  essential  governmental  services, 
emergency  infrastructure  reconstruction,  and  humanitarian  relief’  (JP  3-07,  2011).  While 
both  Iraq  and  Afghanistan  have  been  considered  major  COIN  operations,  the  proper 
doctrine  terms  dictate  that  the  “build”  portion  of  COIN  is  a  stability  operation  (JP  3-07, 
2011).  As  such,  stability  operations  can  consist  of  heavy  humanitarian  assistance  and 
infrastructure  investment  efforts.  The  core  mission  of  stabilization  operations  is  to  build 
the  legitimacy  of  the  host  nation  government  in  the  opinion  of  the  host  nation  population 
(JP  3-07,  2011).  While  U.S.  civilian  agencies  such  as  USAID  and  USACE  should  lead 
the  execution  of  major  infrastructure  projects  to  restore  essential  services  to  the 
population,  U.S.  military  doctrine  specifies  that  military  units  can  be  tasked  to  execute 
quick  impact  projects  as  part  of  a  multi-agency  team  such  as  Provincial  Reconstruction 
Teams  (PRTs).  U.S  military  doctrine  also  cites  CERP  in  Iraq  and  Afghanistan  as  an 
example  of  how  to  implement  economic  development  initiatives  (JP  3-07,  2011). 

Stability  operations  are  intended  to  build  the  legitimacy  of  the  host  nation 
government  as  seen  by  the  host  nation  population,  yet  the  assessment  guidance  in  U.S. 
military  doctrine  emphasizes  Measures  of  Effectiveness  (MOE)  that  measure  the  change 
in  a  citizen’s  life  rather  than  their  perception  of  their  government  and  references  metrics 
such  as  reductions  in  violence  and  improvement  of  public  utility  performance  as 
examples  (JP  3-07,  201 1).  Detailed  assessment  tools  have  been  developed  for  specific 
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assessment  purposes;  for  example,  the  Interagency  Conflict  Assessment  Framework  tool 
is  used  to  determine  the  groups  involved  and  drivers  of  conflict,  whereas  the  District 
Stability  Framework  is  a  tool  used  to  determine  the  root  cause  of  instability  and  identify 
means  to  address  those  causes  (JP  3-07,  201 1).  There  are  four  assessment  tools  listed  in 
JP  3-07,  but  the  only  tool  that  takes  into  account  the  perception  of  the  population  is  the 
District  Stability  Framework. 

COIN  Operations 

Counterinsurgency  (COIN)  operations  have  been  conducted  by  the  U.S.  military 
extensively  in  Iraq,  Afghanistan,  the  Philippines,  and  the  Horn  of  Africa  since  2001. 
COIN  operations  are  a  combined  political  and  military  effort  where  the  legitimacy  of  the 
government  as  viewed  by  the  population  is  the  objective  (JP  3-24,  2013).  JP  3-24  states 
that  the  authority  to  govern  is  dependent  on  four  factors: 

•  Mandate.  The  perceived  legitimacy  of  the  mandate  that  establishes  a  state 
authority,  whether  through  the  principles  of  universal  suffrage,  a  recognized 
or  accepted  caste/tribal  model,  or  authoritarian  rule. 

•  Manner.  The  way  in  which  those  exercising  that  mandate  conduct  themselves, 
both  individually  and  collectively  in  meeting  the  expectations  of  the  local 
population(s). 

•  Support  and  Consent.  The  extent  to  which  local  populations  consent  to,  or 
comply  with,  the  manner/authority  of  those  exercising  the  mandate.  Consent 
may  range  from  active  support,  passive  support,  or  indifference,  through 
unwilling  compliance. 

•  Expectations.  The  relative  quality  or  amount  of  support  that  local  populations 
expect  from  their  government. 

COIN  is  a  comprehensive  approach  to  building  the  legitimacy  of  the  host  nation 
government  through  information,  security,  and  economic  activities  executed  through  a 
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political  strategy  (JP  3-24,  2013).  In  both  Iraq  and  Afghanistan,  the  “Shape,  Clear,  Hold, 
Build,  Transition”  operations  scheme  was  used  for  COIN.  This  method  saw  extensive 
use  of  infrastructure  investment  in  the  “Build”  and  “Transition”  phases  as  part  of  the  U.S. 
military  strategy,  which  could  also  be  considered  stability  operations.  The  security 
activities  of  COIN  secure  the  conditions  for  economic,  political,  and  social  improvements 
by  the  host  nation  government  such  as  restoring  rule  of  law,  providing  essential  services, 
and  rebuilding  economic  systems  (JP  3-24,  2013).  The  complex  relationships  of  these 
factors  resulted  in  the  development  of  a  system  dynamics  diagram  shown  in  Figure  6, 
which  was  leaked  to  the  press  in  2010,  where  General  Stanley  McChrystal,  then 
Commander  of  the  International  Security  Assistance  Force  (ISAF),  commented  “When 
we  understand  that  slide,  we’ll  have  won  the  war”  (Bumiller,  2010). 

Operations  assessments  are  vital  to  every  military  operation,  but  COIN  doctrine 
stresses  that  the  understanding  of  the  COIN  operational  environment  requires  continuous 
assessment  in  order  to  detect  changing  conditions  (JP  3-24,  2013).  JP  3-24  includes  an 
extensive  18-page  long  chapter  dedicated  to  assessing  COIN  operations,  which  is 
considerably  more  than  the  previously  mentioned  doctrine  documents.  While  this  seems 
extensive,  the  operations  assessments  process  is  focused  on  definitions  and  types  of 
inputs  and  outputs  rather  than  suggesting  ways  to  collect  data.  Despite  COIN  being  a 
population-centric  strategy,  there  is  no  specific  mention  of  surveys  of  the  population 
during  operations  assessments. 
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Figure  6:  Afghanistan  Stability  /  COIN  System  Dynamics  Chart  (Bumiller,  2010) 


Operations  Assessments 

According  to  U.S.  military  doctrine,  operations  assessments  should  start  during 
the  initial  planning  stages  of  the  operations  process  and  continue  throughout  execution. 
These  assessments  provide  data  to  commanders  to  determine  progress  towards  defined 
objectives  and  provide  vital  information  for  decisions  (JP  3-0,  201 1).  These  assessments 
use  Measures  of  Effectiveness  (MOE),  which  assess  the  overall  change  in  a  system,  and 
Measures  of  Performance  (MOP),  which  indicate  specific  task  performance  (JP  3-29, 
2014).  While  doctrine  documents  focus  on  defining  these  terms,  they  fall  short  on 
providing  guidance  in  how  to  actually  perform  operations  assessments  (Schroden,  2011). 
As  presented  in  the  COIN  Doctrine  section,  COIN  guidance  does  not  mention  using 
population  surveys  to  determine  their  perception  of  their  own  government. 

Assessments  of  whether  the  U.S.  military  operations  in  Iraq  and  Afghanistan  were 
winning  or  losing  was  difficult  to  accomplish  and  widely  criticized  (Schroden,  2011). 
Some  critics  have  cited  improper  metrics,  but  Schroden  identified  five  factors  in  his 
“Failure  Cycle  for  Operations  Assessment,”  with  poor  doctrine,  inadequate  training,  poor 
processes  and  products,  commander  disinterest,  and  lack  of  advocacy  listed  as  the  links  in 
a  chain  causing  failure. 

Poor  and  confusing  doctrine  leads  (in  part)  to  inadequate  (or  no)  training 
of  assessment  practitioners,  which  leads  to  poor  assessment  processes  and 
products,  which  leads  to  commanders  who  are  uninterested  in  assessment, 
which  leads  to  a  lack  of  advocacy  for  fixing  assessment,  which  leads  to  a 
perpetuation  of  poor  doctrine  -  and  the  cycle  continues.  (Schroden,  2011) 

In  the  case  of  infrastructure  investment  in  both  Iraq  and  Afghanistan,  it  could  be  argued 

that  doctrine  did  not  provide  adequate  assessment  guidance  since  COIN  does  not  mention 

population  surveys  despite  being  a  population-centric  strategy.  The  lack  of  this  data  left 
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a  key  variable  unmeasured  in  COIN  assessment  such  that  training  of  the  assessors  may 
have  been  irrelevant,  but  the  omission  of  population  perception  survey  data  likely 
resulted  in  an  assessment  product  that  did  not  measure  the  progress  towards  the  ultimate 
objective  in  a  COIN  operation. 

Operations  assessments  are  not  limited  to  combat  operations.  By  law,  the  U.S. 
military  must  conduct  an  after-action  review  or  project  evaluation  on  all  OHDACA 
projects  within  30  days  of  project  completion  and  again  a  year  after  completion;  however, 
from  FY2005  to  FY2009,  53%  of  surveyed  projects  did  not  have  a  30-day  evaluation 
accomplished  and  over  90%  did  not  have  a  1-year  evaluation  (GAO,  2012).  The  GAO 
also  criticized  the  U.S.  military  for  using  inconsistent  evaluation  of  HCA  projects  and  for 
executing  projects  that  did  not  meet  host  country  needs  (GAO,  2012). 

Impact  of  Humanitarian  Assistance  and  Foreign  Aid 

The  impact  of  humanitarian  assistance  and  foreign  aid  is  a  controversial  subject. 
Core  to  the  assessment  of  the  impact  of  humanitarian  assistance  or  foreign  aid  is  defining 
the  intended  effect.  During  the  Cold  War,  the  U.S.  used  foreign  aid  as  a  tool  to  spread 
democracy,  but  little  evidence  exists  to  prove  that  this  aid  had  a  significant  impact  on 
recipient  nations’  progress  towards  democracy.  Analysis  indicates  that  progress  towards 
democracy  is  often  offset  by  a  country’s  dependence  on  aid  which  also  results  in  a 
government  becoming  less  accountable  to  their  citizens  but  rather  to  their  aid  donors 
(Knack,  2004).  Others  argue  that  variables  such  as  the  style  of  government  and  the 
breadth  of  distribution  of  wealth  within  a  country  influence  progress  towards 
democratization  upon  a  dictator  receiving  foreign  aid  (Wright,  2009). 
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Several  studies  have  been  conducted  studying  the  effect  of  CERP  spending  in  Iraq 
and  Afghanistan  on  violence  levels.  An  analysis  of  2,000  CERP  records  in  the 
Afghanistan  project  database  reported  less  than  10%  of  those  projects  identified  violence 
reduction  as  an  intended  outcome  (Fischerkeller,  2011).  Many  respondents  to  a  survey 
conducted  by  SIGIR  of  personnel  involved  with  CERP  in  Iraq  reported  that  reduction  in 
violence  was  the  intended  impact;  however,  the  intended  impact  of  a  project  such  as 
“increase  government  capacity”  was  often  not  adequately  related  to  the  metrics  used  to 
assess  the  impact  or  to  the  actual  project  executed  (Bowen  &  Collier,  2013). 

Data  from  Iraq  supported  a  model  for  determining  the  impact  of  CERP  project 
spending  on  violence  levels,  which  indicated  that  “every  additional  dollar  per  capita  of 
CERP  spending  predicted  1.59  less  violent  incidents  per  100,000  population  per  half 
year,”  (Berman,  Shapiro,  Felter,  2011).  The  data  also  supports  the  conclusion  that 
reconstruction  spending  outside  of  CERP,  which  accounted  for  approximately  90%  of  the 
funds  expensed  over  the  analyzed  time  period,  did  not  have  a  violence-reducing  effect 
(Berman,  Shapiro,  Felter,  201 1).  Similar  results  in  another  study  show  that  the  type  of 
project  executed  and  a  city’s  baseline  violence  level  in  Iraq  influence  whether  or  not 
spending  through  CERP  reduces  violence  levels.  While  many  of  the  results  were  not 
statistically  significant  at  the  p  =  0.01  level,  the  data  shows  that  spending  on  democracy, 
education,  transportation,  and  water,  as  well  as  large  projects  in  general,  reduced  the  per 
capita  violence  rates  in  cities  with  a  higher  baseline  violence  level;  however,  spending  in 
lower  violence  baseline  cities  on  democracy,  transportation,  and  water  were  the  only  ones 
that  reduced  violence  (Clark  &  Jackson,  2013).  The  results  indicate  that  smaller  projects 
executed  at  lower  levels  were  more  effective  at  reducing  violence  in  some  situations,  but 
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it  is  unclear  if  the  larger  projects  had  an  impact  on  other  aspects  of  COIN  and  stability 
objectives  such  as  building  the  legitimacy  of  the  Iraqi  government. 

Recent  research  into  stability  and  international  aid  is  inconclusive  on  the  overall 
effect  of  assistance  in  Afghanistan.  In  one  study,  evidence  suggests  that  the  large 
amounts  of  aid  being  spent  enabled  corruption  and  decreased  the  perception  of  legitimacy 
of  the  government  and,  in  some  situations,  actually  increased  violence  as  parties 
competed  for  aid  resources  (Fishstein  &  Wilder,  2012).  In  COIN  and  stability  operations, 
the  objective  is  to  increase  the  legitimacy  of  the  government,  yet  few  studies  address  how 
aid  may  impact  this  outcome.  Measuring  the  impact  of  aid  is  difficult,  but  public  opinion 
polling  may  be  a  method  that  has  merit.  In  many  cases,  the  impact  of  PRTs  has  been 
reported  in  terms  of  dollars  spent  and  numbers  of  projects  executed;  in  other  cases,  the 
metrics  of  success  have  been  reported  in  terms  of  number  of  smiling  children  (McNerney, 
2006).  Fishstein  and  Wilder  (2012)  caution  that  western- style  polls  may  be  unreliable; 
however,  organizations  should  invest  more  in  measuring  outcomes  and  impacts  of  aid. 

Citizen  Satisfaction  Research  Background 

The  discipline  of  Public  Administration  focuses  in  the  area  of  citizen  satisfaction 
and  has  mature  models  developed  to  measure  citizen  satisfaction  with  government 
services,  along  with  a  multitude  of  other  measures  such  as  justice  and  trust.  Within  the 
discipline  of  Public  Administration,  there  has  been  a  recent  synthesis  of  traditional  citizen 
satisfaction  theory  with  consumer  expectancy-disconfirmation  theory  that  originated  in 
the  marketing  and  business  fields  (Van  Ryzin  et  al.,  2004).  This  model  has  been  adapted 
and  tested  by  Van  Ryzin,  who  has  published  several  articles  with  his  results.  The  core  of 
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his  adapted  expectation,  performance,  and  citizen  satisfaction  model  compares  a  citizen’s 
survey  ratings  of  the  level  of  performance  of  nine  city  services  with  their  previous 
expectations  of  level  of  perfonnance,  and  then  compares  this  relationship  to  the  citizen’s 
overall  satisfaction  with  government  services  (Van  Ryzin,  2004). 

Van  Ryzin’s  model  as  represented  in  Figure  7  illustrates  that  citizen  expectations 
of  government  services  and  perceptions  of  performance  feed  into  the  overall  perception 
of  disconfinnation.  The  citizen’s  perceived  difference  between  their  expectations  and 
perceived  performance  will  either  be  positive,  meaning  government  services  exceeded 
their  expectations,  or  will  be  negative,  meaning  government  services  did  not  meet  their 
expectations.  This  disconfinnation  variable  then  drives  the  citizen’s  overall  perception  of 
satisfaction  (Van  Ryzin,  2004). 


Expectations  j 


l  Disconfinnation  V 


Satisfaction 


B 


Performance 


Figure  7:  Expectancy  Disconfinnation  Model  (Van  Ryzin,  2004) 
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Van  Ryzin’s  first  model  showed  a  strong  relationship  between  disconfirmation 
and  satisfaction;  however,  later  results  determined  that  the  initial  model  had  statistical 
flaws  which  biased  the  results  (Van  Ryzin,  2005).  These  flaws  were  identified  and 
modified  in  later  studies  by  measuring  disconfirmation  with  a  survey  response  instead  of 
calculating  disconfirmation  by  subtracting  the  expectations  rating  from  the  performance 
rating.  A  new  model  was  tested  by  Van  Ryzin  using  an  online  survey  sent  to  1,631 
people,  of  which  615  people  completed  the  survey  (Van  Ryzin,  2005).  The  results  of  this 
survey  formed  the  basis  of  his  2005  paper  which  tested  three  similar  models,  of  which  the 
most  applicable  model  appeared  to  be  one  with  a  survey  question  response  for 
disconfirmation.  This  differed  from  the  other  two  models  which  used  a  mathematical 
disconfirmation  value,  and  the  combination  of  a  mathematical  value  and  survey  data,  to 
determine  citizen  satisfaction.  Using  the  survey-only  measure  for  disconfirmation 
reduced  the  effect  of  disconfirmation  on  satisfaction,  but  it  still  accounted  for  a 
standardized  effect  of  0.49  on  the  overall  satisfaction  rating,  with  performance 
contributing  0.41  and  expectations  contributing  0.10  towards  this  measure  as  well.  The 
summary  model  of  these  results  is  included  in  Figure  8,  which  illustrates  that  the  total 
model  explained  75%  of  the  variation  of  the  data  (Van  Ryzin,  2005). 
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Figure  8:  Expectancy  Disconfirmation  Model  Parameter  Estimates  (Van  Ryzin,  2005) 

Using  this  framework,  the  impact  of  humanitarian  assistance  can  be  assessed  by 
measuring  the  change  in  citizen  satisfaction  before  and  after  a  humanitarian  project  is 
executed.  Van  Ryzin’s  model  was  developed  from  research  on  government  services  in 
New  York  City,  which  will  require  modification  of  the  measured  factors  when  used  in  a 
less  developed  country,  as  fewer  government  services  are  typically  available  to  such  a 
population.  Hansen  (2015)  adapted  this  framework  to  measure  citizen  satisfaction  in 
Belize,  which  formed  the  basis  for  this  research. 
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Background  on  Belize 

Belize  is  a  former  British  Colony  in  Central  America.  Both  the  British  and 
Spanish  claimed  Belize  until  1854  when  it  was  designated  British  Honduras,  and  Belize 
was  granted  independence  from  the  United  Kingdom  in  1991.  The  current  population  of 
Belize  is  approximately  340  thousand  people  and  the  country’s  gross  domestic  product 
(GDP)  is  approximately  $1.6  billion  per  year,  which  yields  a  per  capita  GDP  of  $8.8 
thousand  per  citizen  (CIA:  The  World  Factbook,  2014).  The  largest  sector  of  Belize’s 
economy  is  tourism,  which  accounts  for  33.2%  of  the  GDP  and  30.1%  of  the  employed 
workforce  (World  Travel  &  Tourism  Council,  2012).  The  rest  of  Belize’s  economy 
consists  of  agricultural  exports  such  as  bananas,  cacao,  citrus,  sugar,  lumber,  and  fish,  as 
well  as  industries  such  as  textile  manufacturing,  oil,  and  food  processing  (CIA:  The 
World  Factbook,  2014).  Figure  9  shows  a  map  of  Belize  that  illustrates  locations  of 
interest  and  the  geography  of  the  country.  Overall,  the  country  is  impoverished  with  an 
economy  based  on  services,  exporting  resources,  and  limited  industrial  capacity. 
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Figure  9:  Map  of  Belize  (belize.com,  2015) 
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In  contrast  to  these  average  ratings  of  government  trust,  the  average  rating  of 
Belize  citizens’  satisfaction  with  municipal  services  in  2008  was  the  third  lowest  in  Latin 
America  with  a  39.6  satisfaction  rating.  The  average  rating  was  49.9  elsewhere  in  Latin 
America,  and  the  only  countries  lower  than  Belize  in  that  study  were  Haiti  and  Jamaica, 
at  39.5  and  37,  respectively  (Montalvo,  2009).  Similar  results  can  be  seen  over  time  from 
2008  through  2012  in  Figure  10,  which  was  compiled  using  data  from  LAPOP’s  System 
for  Online  Data  Analysis.  This  chart  illustrates  that  less  than  20%  of  Belize  citizens  have 
a  favorable  rating  of  municipal  services;  low  ratings  are  consistent  across  several  years, 
and  the  percentage  of  people  rating  their  municipal  services  “Poor”  or  “Very  Poor”  has 
decreased  in  the  last  4  years  (LAPOP,  2014).  These  data  suggest  that  the  overall  level  of 
municipal  services  are  lacking  in  Belize  compared  to  the  rest  of  Latin  America,  but 
citizens’  trust  in  government  is  similar  to  other  countries  in  the  region. 
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Figure  10:  Citizen's  Perceptions  of  Local  Government  Services  in  Belize,  2008-2012 

(Adapted  from  LAPOP,  2014) 


Another  area  of  interest  for  this  research  is  the  population’s  perception  of  the  U.S. 
Recent  data  indicates  that  the  perception  of  trust  in  the  U.S.  among  citizens  of  Belize  is 
rated  48.9  out  of  100,  which  is  below  the  Latin  America  average  of  55.6.  Trust  in  the 
U.S.  was  analyzed  using  regression  models  which  determined  that  trust  ratings  are  driven 
by  trade  with  the  U.S.,  and  many  of  the  countries  with  the  lowest  trust  ratings  in  the  U.S. 
were  located  farther  away  than  countries  with  higher  ratings,  with  the  exception  of 
Mexico  (Silliman,  2014).  While  the  citizens  of  Belize  have  a  relatively  low  level  of  trust 
in  the  U.S.  despite  being  geographically  close,  it  should  also  be  noted  that  the  rating  of 
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trust  in  the  U.S.  of  48.9  is  higher  than  the  citizens  of  Belize’s  rating  of  47.2  regarding 
trust  in  their  own  government. 

In  addition  to  these  measures  of  government  services  and  trust,  there  is  data 
available  regarding  overall  life  satisfaction  within  Latin  America.  Like  many  of  the  other 
measures  previously  discussed,  Belize  rated  only  53.4  out  of  100,  below  the  average 
rating  of  59.5  for  Latin  America,  with  only  four  other  countries  rating  lower  than  Belize, 
and  with  Brazil  ranked  highest  at  71.6  in  2010.  Further  analysis  suggests  a  link  between 
individual  wealth,  national  wealth,  and  happiness  ratings  (Corral,  2011).  These  data 
appear  to  show  a  different  picture  than  the  data  regarding  trust  in  government,  which 
showed  ratings  close  to  the  average  for  Latin  America,  although  that  data  was  correlated 
to  GDP  growth  rate  per  capita.  Further  analysis  regarding  the  determinants  of 
satisfaction  with  government  services,  trust  in  government,  and  overall  happiness  is 
warranted  to  better  understand  these  relationships. 

Operation  NEW  HORIZONS  14 

The  820th  RED  HORSE  Squadron  (RHS)  deployed  to  Belize  from  April  through 
June  2014  in  support  of  Operation  NEW  HORIZONS  14.  During  this  deployment,  the 
820th  RHS  executed  five  construction  projects  at  three  locations  which  had  been  selected 
prior  to  this  research  effort.  These  projects  ranged  in  cost  from  $75  thousand  to  $204 
thousand  each.  Projects  included  four  school  additions  and  one  hospital  addition,  with 
three  school  projects  occurring  in  Belize  City  and  the  other  projects  occurring  in 
Belmopan  and  Hattieville.  A  summary  of  these  projects  and  locations  is  included  in 
Table  1.  This  operation  provided  an  opportunity  to  gather  population  survey  data  before 
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and  after  the  execution  of  these  projects  in  order  to  determine  the  effect  of  the  project  on 


the  population’s  satisfaction  with  their  host  nation  (HN)  government  and  their  perception 
of  the  U.S. 


Table  1:  Operation  NEW  HORIZONS  14  Project  Summary 


Project  Title 

Location 

Cost 

Western  Regional  Hospital  Addition 

Belmopan,  Belize 

$204,000 

Hattieville  Preschool 

Hattieville,  Belize 

$173,000 

Sadie  Vernon  School 

Belize  City,  Belize 

$144,000 

Stella  Maris  School 

Belize  City,  Belize 

$119,000 

Edward  P.  York  School 

Belize  City,  Belize 

$75,000 

Experiment  Design 

While  both  Hansen  and  this  research  effort  answer  distinctly  different  questions, 
the  core  research  effort  shared  common  survey  data  collected  during  Operation  NEW 
HORIZONS  14  in  Belize.  An  experiment  was  designed  to  test  the  effect  of  a  given 
humanitarian  project  on  a  population.  The  treatment  for  this  experiment  is  the  project 
being  executed  in  a  given  community.  Because  treatments  are  being  executed  on  existing 
groups  as  the  community  population,  a  quasi-experimental  approach  using  the 
nonequivalent  control  group  design  was  used  to  test  the  hypotheses  (Patten,  2009).  A 
diagram  of  this  experimental  design  is  included  in  Figure  11,  with  the  four  school 
projects  and  one  clinic  project  consisting  of  the  three  treatments  in  this  diagram,  and  a 
sixth  control  group  at  the  bottom.  Since  three  of  the  school  projects  were  executed  in 
Belize  City,  the  treatments  and  observations  were  combined.  The  dashed  lines  indicate 
that  these  treatments  are  being  done  on  existing  separate  populations  of  Belmopan, 
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Hattieville,  Belize  City,  and  control  locations.  Surveys  were  given  before  and  after  the 
treatments  to  determine  the  population’s  change  in  satisfaction  with  government  services 
and  perception  of  the  U.S.  due  to  the  infrastructure  investment. 
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X 
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Control 
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0 

Figure  1 1 :  Experiment  Design 


Survey  Development 

The  population’s  satisfaction  with  government  services  was  measured  with  a 
survey  using  the  expectancy-disconfirmation  model  proposed  by  Van  Ryzin  (2004).  The 
structure  of  Van  Ryzin’ s  survey  instrument  (2005)  was  used  as  a  basis  to  develop  the 
survey  instrument  employed  at  multiple  locations  across  Belize,  with  additional  questions 
included  to  measure  factors  not  considered  in  Van  Ryzin’s  research.  The  survey  first 
measured  the  citizen’s  top  three  priority  services  in  a  free  response  format  to  determine 
the  citizen’s  priority  of  services  without  being  prompted  by  a  Likert  scale  question.  Next, 
the  survey  collected  ratings  of  expectations,  performance,  disconfirmation,  and 
satisfaction  with  government  services  on  a  1  to  5  Likert  scale,  with  three  questions  for 
each  measure  except  performance,  which  had  seven  measures. 

The  seven  performance  measures  asked  citizens  to  rate  the  performance  of 
schools,  health  care,  local  police,  garbage  removal,  quality  of  drinking  water,  cleanliness, 


34 


and  quality  of  roads  in  their  neighborhood.  These  measures  were  selected  based  on  the 
public  services  available  in  Belize.  The  next  section  of  the  survey  included  a  series  of 
questions  designed  to  assess  the  quality  of  the  target  selected  in  that  area  for  diagnostic 
purposes  and  asked  questions  unrelated  to  infrastructure  to  determine  if  there  is  an 
external  threat  to  validity  on  the  results  of  the  satisfaction  rating.  The  next  section  asked 
citizens  about  their  perceptions  of  the  U.S.  and  U.S.  military  personnel  in  Belize  during 
the  exercise.  Finally,  the  survey  concluded  with  date,  location,  and  demographic 
questions  to  determine  if  the  number  of  children,  education  level,  income  level,  gender, 
or  age  have  an  effect  on  the  results.  A  summary  of  the  survey  questions  is  included  in 
Table  2;  the  survey  instrument  itself  is  included  in  Appendix  A. 

Latent  variables  in  the  expectancy-disconfirmation  measures  and  the  perceptions 
of  the  U.S.  measure  have  multiple  survey  questions  to  improve  the  reliability  of  the 
survey  data.  For  the  expectations,  disconfirmation,  and  satisfaction  measures  there  are 
three  questions  each,  while  the  perceptions  of  the  U.S.  measure  has  four  questions.  The 
citizen  priority  measure  has  three  questions  and  the  services  measure  has  eight  questions. 
However,  each  project  will  only  influence  one  of  these  measures;  therefore,  it  should  not 
be  considered  a  reliability  improvement  for  having  multiple  questions.  Similarly,  the 
target  quality  questions  are  all  specific  to  aspects  of  the  target  such  as  the  infrastructure, 
staff,  and  equipment  of  these  services.  These  measured  served  as  a  diagnostic  to 
determine  what  aspects  of  the  service  are  better  in  their  opinion,  and  will  better  explain 
the  changes  before  and  after  the  infrastructure  improvement  is  executed  at  that  location. 
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Table  2:  Survey  Question  Structure 


Section 

Measure 

Type  of 
Measure 

Number  of 
Questions 

Section  1 :  Priorities 

Citizen’s  Priority  of  Services 

Free  Response 

3 

Section  2:  Expectancy  - 
Disconfirmation  Series 

Expectations 

1-5  Likert 

3 

Performance 

1-5  Likert 

8 

Disconfirmation 

1-5  Likert 

3 

Satisfaction 

1-5  Likert 

3 

Section  3:  Target 

Quality 

Factors  of  Target  Quality 

1-5  Likert 

4 

Section  4:  Perception 
of  US 

Citizen’s  Perception  of  US 
due  to  HA  in  Area 

1-5  Likert 

4 

Section  5: 

Demographics 

Demographics,  Location, 
Date 

Free  Response 

7 

Survey  Procedure 

Survey  data  was  collected  by  Belize  Defense  Forces  (BDF)  personnel  from  the 
population  before  and  after  each  of  the  five  projects  was  executed.  The  official  language 
of  Belize  is  English,  which  negated  the  requirement  for  language  translation  and  back- 
translation  of  the  survey  instrument;  however,  due  to  a  literacy  rate  of  only  76.8%  (CIA: 
The  World  Factbook,  2014),  the  survey  was  designed  to  be  read  to  the  respondent  for 
verbal  answers  to  questions  with  BDF  personnel  circling  answers  on  paper  surveys.  The 
survey  results  were  compared  to  determine  the  strength  of  the  statistical  difference 
between  satisfaction  ratings  as  well  as  multiple  regression  analysis  to  determine  the  effect 
of  the  project  on  the  population’s  satisfaction.  According  to  Cohen  (1992)  and 
anticipating  medium-strength  effect  size  for  three  multiple  regression  factors  and  an 
alpha  of  0.05,  the  estimated  minimum  number  of  samples  per  location  was  76  both  before 
and  after  each  project  plus  a  control  set,  for  a  minimum  total  number  of  912  samples  for 
the  entire  project. 
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Anticipated  Significance 

This  research  is  intended  to  develop  further  understanding  of  citizen  satisfaction 
with  government  services  that  may  be  applied  towards  a  target  selection  methodology  for 
humanitarian  assistance  projects.  This  research  may  also  benefit  municipal  governments 
by  contributing  to  the  body  of  knowledge  by  further  exploring  the  influence  of 
government  service  performance  on  citizen  satisfaction. 

Overview  of  Remaining  Chapters 

This  thesis  is  in  the  scholarly  article  format.  The  following  chapter  is  an  article 
that  contains  all  of  the  elements  of  a  thesis;  it  could  be  submitted  for  publication  when 
combined  with  the  article  written  by  Hansen.  As  an  independent  chapter,  it  includes  an 
abstract,  introduction,  background,  methodology,  and  discussion  section.  Chapter  3 
concludes  this  thesis  with  a  discussion  of  the  results  and  conclusions;  it  also  proposes 
further  research  opportunities  within  this  topic. 
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II.  Scholarly  Article 


Abstract 

Assessing  the  effectiveness  of  humanitarian  assistance  and  foreign  aid  has 
presented  a  challenge  to  personnel  tasked  with  these  operations.  Answering  the  question 
“are  we  winning  hearts  and  minds”  has  similarly  eluded  military  personnel  in  Iraq  and 
Afghanistan.  This  research  presents  a  review  of  humanitarian  assistance  and  foreign  aid 
conducted  by  the  U.S.  government  and  literature  on  U.S.  military  doctrine  regarding 
humanitarian  assistance  and  infrastructure  investment.  The  research  builds  upon 
expectancy  disconfirmation  theory  research  to  determine  the  strongest  predictors  of 
citizen  satisfaction  with  government  services.  Case  study  data  was  collected  in  Belize 
before  and  after  a  U.S.  military  humanitarian  and  civil  assistance  construction  project  was 
executed,  and  this  data  was  analyzed  using  an  expectancy  theory  model.  The  results 
indicate  that  the  model  using  performance,  disconfirmation,  and  an  interaction  effect  of 
both  explains  56%  of  the  variation  in  citizen  satisfaction  and  proposes  a  predictive  model 
of  citizen  satisfaction.  This  article  proposes  further  research  with  improvement  to  the 
survey  methods  and  instrument;  it  also  discusses  how  the  model  may  not  account  for  an 
unmeasured  variable.  Further  research  is  also  suggested  to  determine  the  relationship  of 
time  and  location  on  a  citizen’s  satisfaction  rating  when  considering  the  impact  of  a 
humanitarian  project. 
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Introduction 


The  United  States  entered  the  domain  of  major  world  aid  provider  starting  with 
the  Marshall  Plan  following  World  War  II.  Since  then,  the  United  States  has  been 
heavily  involved  in  foreign  aid  and  humanitarian  assistance  to  present  day.  These  aid 
programs  are  often  executed  by  the  Department  of  State,  but  the  U.S  military  has  taken 
on  enormous  aid  programs  since  the  invasions  of  Afghanistan  and  Iraq  in  2001.  While 
the  goals  and  means  of  these  programs  vary  considerably,  all  share  a  common  deficiency 
in  assessing  the  impact  of  these  aid  efforts.  This  article  will  discuss  means  of  U.S. 
humanitarian  assistance  and  foreign  aid,  methods  of  assessing  the  effectiveness  of  aid, 
and  will  propose  an  adaptation  of  existing  citizen  satisfaction  measures  to  analyze  the 
relationship  of  performance  and  disconfirmation  on  citizen  satisfaction.  Results  of  these 
analyses  will  be  presented  and  discussed,  and  further  research  efforts  are  proposed. 

Background 

The  U.S.  military  has  statutory  authority  to  conduct  foreign  assistance  through 
two  major  appropriations:  Overseas  Humanitarian,  Disaster,  and  Civic  Aid  (OHDACA) 
and  Humanitarian  and  Civic  Assistance  (HCA).  OHDACA  is  primarily  used  for  disaster 
response  operations,  with  the  military  spending  $328.4  million  between  2005  and  2010. 
By  contrast,  HCA  is  used  for  troop  deployment  training  exercises  in  foreign  countries 
where  medical  assistance  or  construction  projects  are  executed,  with  $75.1  million  being 
spent  between  2005  and  2010  at  various  locations  around  the  world  (GAO,  2012). 

Following  the  invasions  of  Afghanistan  and  Iraq,  the  U.S.  military  undertook  the 
much-criticized  Commander’s  Emergency  Response  Program  (CERP).  This  program 
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originated  in  Iraq  and  was  initially  executed  with  seized  Iraqi  Ba’ath  Party  funds  on 
emergency  humanitarian  needs;  however,  once  the  Iraqi  funds  were  exhausted,  this 
program  was  then  funded  with  U.S.  appropriated  funds  (Martins,  2005).  By  the  end  of 
the  Iraq  War  in  2011,  the  U.S.  military  had  executed  over  36,000  projects  at  a  total  cost 
of  $3.7  billion  (SIGIR,  2013a).  A  similar  CERP  effort  was  started  in  Afghanistan  in 
2004,  with  the  key  difference  being  that  this  program  was  always  resourced  through 
appropriated  funds,  with  $3.7  billion  spent  in  Afghanistan  through  July  2014  (SIGAR, 
2014). 

While  CERP  represents  a  large  amount  of  aid,  this  total  is  eclipsed  by  the  amount 
of  non-military  humanitarian  assistance  programs  executed  by  other  U.S.  agencies.  The 
U.S.  government  spent  $25.7  billion  in  Iraq  from  2003  through  2011,  with  $20.9  billion 
spent  on  critical  infrastructure  and  $4.8  billion  spent  on  economic  development  (SIGIR, 
2013a).  In  Afghanistan,  the  U.S.  has  spent  $19.5  billion  through  2014,  with  $17.5  billion 
spent  on  economic  development,  $1.2  billion  on  large  public  infrastructure,  and  $0.8 
billion  on  banking  sector  reforms  (SIGAR,  2014).  These  programs  were  executed  mainly 
through  U.S.  Agency  for  International  Development  (USAID),  but  other  agencies  such  as 
the  Departments  of  State,  Agriculture,  Justice,  Commerce,  and  Treasury  have  all 
participated  as  well. 

Humanitarian  assistance  and  foreign  aid  have  been  used  in  enormous  quantities  to 
support  the  U.S.  military’s  counterinsurgency  (COIN)  campaigns  in  Iraq  and 
Afghanistan.  The  ultimate  objective  according  to  U.S  military  doctrine  is  to  build  the 
legitimacy  of  the  host  nation  government  in  the  opinion  of  the  population  (JP  3-24, 

2013).  This  is  accomplished  through  providing  security  to  the  population  in  the  “Clear” 
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and  “Hold”  phases  of  operations,  which  sets  the  stage  for  the  “Build”  and  “Transition” 
phases  to  start  with  heavy  investment  in  essential  government  services,  economic 
systems,  and  infrastructure  (JP  3-24,  2013).  Even  though  the  stated  goal  of  these 
operations  is  to  build  legitimacy  of  a  government  as  perceived  by  their  constituents, 
CERP  projects  were  often  assessed  based  on  measuring  the  change  in  levels  of  violence 
(Bowen  &  Collier,  2013)  despite  less  than  10%  of  projects  being  identified  as  intended  to 
reduce  violence  (Fisherkeller,  2011). 

Stability  operations  are  similar  to  the  “Build”  and  “Transition”  phases  of  a  COIN 
operation  with  a  common  goal  of  building  the  legitimacy  of  the  host  nation  government 
and  are  often  accompanied  by  humanitarian  assistance  and  infrastructure  investment  (JP 
3-07,  201 1).  These  projects  have  been  executed  through  CERP  and  other  humanitarian 
and  foreign  aid  programs  in  Iraq,  Afghanistan,  and  in  various  locations  worldwide,  but 
most  of  the  data  and  research  is  from  the  conflicts  in  Iraq  and  Afghanistan.  U.S.  military 
doctrine  recommends  using  assessment  tools  such  as  USAID’s  District  Stability 
Framework,  which  uses  a  variety  of  analysis  and  survey  data  collection  to  conduct  root 
cause  analysis  of  sources  of  instability  and  address  them  accordingly  (JP  3-07,  2011). 
This  tool  can  be  used  to  measure  population  perceptions  of  the  legitimacy  of  their 
government,  but  implementation  of  this  process  occurred  late  in  the  Afghan  campaign. 

Assessing  the  impact  of  humanitarian  projects  in  peacetime,  COIN,  and  stability 
operation  environments  is  difficult  and  has  been  criticized,  with  some  examples  citing 
“number  of  smiling  Afghan  children”  as  a  measure  of  effectiveness  (McNerney,  2006). 
Research  in  this  area  has  focused  on  determining  the  effect  of  CERP  spending  on 
violence  levels.  In  one  study,  the  per  capita  spending  on  projects  was  linked  to  reducing 
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the  violence  rate  in  Iraq  over  the  studied  period  (Berman,  Shapiro,  Felter,  2011),  while 
another  study  linked  the  effect  of  specific  project  types  such  as  education  and  water 
improvements  to  different  effect  magnitudes  and  directions  on  violence  rates  at  a  city 
level  (Clark  &  Jackson,  2013).  While  the  reduction  in  violence  is  certainly  a  metric 
worthy  of  measurement  and  analysis  in  a  military  operation,  it  is  unlikely  that  this  is  the 
best  metric  for  assessing  progress  towards  building  the  legitimacy  of  the  host  nation 
government. 

Similarly,  assessing  the  impacts  of  peacetime  foreign  aid  is  difficult  and 
controversial.  Between  2005  and  2009,  53%  of  executed  OHDACA  projects  were  not 
assessed  within  a  month  of  completion  and  90%  of  projects  were  not  assessed  a  year 
later.  Additionally,  the  GAO  criticized  the  lack  of  measures  of  effectiveness  for  HCA 
progress,  noting  examples  such  as  claims  that  “vaccinating  cattle  in  Uganda  helps 
counterterrorism  efforts  in  Somalia”  without  evidence  of  this  causation  link  (GAO, 

2012).  The  GAO  recommended  implementing  RAND’s  monitoring  and  evaluation 
process  for  humanitarian  assistance  projects;  however,  this  guidance  sidesteps  the  key 
question  of  how  to  measure  if  a  project  was  effective  at  large-level  impacts,  but  rather 
focuses  on  project- specific  outputs  and  measures  of  performance  scored  by  personnel 
executing  the  project  (Haims  et  al.,  2011). 

Project  assessments  are  much  more  difficult  when  the  objective  is  to  influence  a 
population’s  perception  of  their  government.  Some  HCA  operations  have  objectives  to 
gain  influence  of  a  host  nation  and  build  relationships,  but  evaluations  fall  short  of 
proving  these  goals  were  achieved  (GAO,  2012).  Other  studies  are  inconclusive  on  the 
impact  of  foreign  aid;  some  authors  have  argued  that  providing  aid  fosters  dependence  on 
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aid  and  reduces  the  accountability  of  a  government  to  their  population  (Knack,  2004), 
while  others  argue  that  the  overall  result  of  foreign  aid  may  result  in  shifting  countries 
from  dictatorships  to  democracy  only  if  certain  conditions  exist  (Wright,  2009).  A  study 
from  Afghanistan  suggests  that  U.S.  aid  not  only  decreased  the  perception  of  legitimacy 
of  the  government  in  some  areas,  but  it  also  increased  corruption  and  violence  in  the 
province  as  parties  feuded  over  the  funds  being  distributed  for  projects  (Fishstein  & 
Wilder,  2012). 

The  National  Solidarity  Programme  (NSP)  conducted  an  extensive  survey  of  the 
impact  of  their  aid  program  at  500  locations  within  Afghanistan  between  2007  and  2011. 
This  study  concluded  that  infrastructure  projects  increase  access  to  utilities  and  improve 
government  services  for  women,  but  irrigation  and  transportation  projects  do  not  have  a 
significant  impact.  The  study  further  states  that  NSP  did  not  have  an  impact  on  economic 
development  and  that  the  quality  of  local  governance  declines  as  the  power  in  the  village 
shifts  from  the  government  to  the  NSP  administration  structure.  Finally,  the  study  also 
concluded  that  NSP  temporarily  increases  the  perception  of  legitimacy  of  the  government 
during  project  execution,  but  that  perception  resets  after  funds  are  expended  (Beath, 
Christia,  Enikolopov,  2013).  The  study  reinforces  the  notion  that  donors,  while  well- 
intentioned  and  executing  quality  projects  that  benefit  people,  may  unintentionally 
subvert  an  existing  government  structure  they  intended  to  assist. 

Despite  extensive  experience  in  humanitarian  assistance  since  2001,  the  U.S. 
military  still  struggles  to  assess  the  effectiveness  of  assistance  in  building  the  legitimacy 
of  host  nation  governments.  There  are  limited  examples  of  personnel  conducting  surveys 
of  a  population  to  determine  their  perceptions  of  their  government  and  public  services, 
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but  many  of  these  surveys  lack  quantitative  elements  for  analysis.  The  absence  of  these 
quantitative  measures  has  made  assessing  the  effectiveness  of  humanitarian  assistance 
difficult  and  lacking  in  rigorous  analytical  power. 

Citizen  satisfaction  models  present  an  area  to  explore  for  assessing  effectiveness 
of  humanitarian  assistance.  A  citizen’s  satisfaction  with  their  government  may  be  an 
appropriate  measurement  to  gauge  the  effectiveness  of  humanitarian  assistance  when  the 
assistance  is  in  the  form  of  an  improvement  to  a  government  service  or  public 
infrastructure.  A  model  of  citizen  satisfaction  has  been  developed  as  an  adaptation  of 
consumer  expectancy  theory,  from  the  marketing  and  business  disciplines,  to  explain 
citizen  satisfaction  (Van  Ryzin  et  al.,  2004).  The  model  measures  a  citizen’s 
expectations,  performance  rating,  and  overall  satisfaction  with  government  services,  as 
well  as  their  perceived  difference  between  their  expectations  and  the  quality  of  the 
service,  which  is  termed  disconfirmation.  Figure  12  shows  Van  Ryzin’s  model  (Van 
Ryzin,  et  al.,  2004). 

Subsequent  studies  and  tests  have  confirmed  its  use  in  urban  U.S.  cities.  One  of 
the  key  conclusions  is  that  a  survey  question  rating  of  disconfirmation  is  a  better  method 
for  measuring  disconfirmation  rather  than  a  function  of  the  difference  between 
expectations  and  performance  as  the  result,  with  the  model  explaining  75%  of  the 
variance  in  the  model  through  a  structural  equation  modeling  analysis  (Van  Ryzin,  2005). 
The  strength  of  this  model  presents  this  as  a  possible  method  to  assess  the  impact  of 
humanitarian  projects  executed  with  an  intended  goal  of  increasing  a  population’s 
perception  of  their  host  nation  government. 
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Figure  12:  Citizen  Satisfaction  Expectancy  Disconfirmation  Model  (Van  Ryzin  et  al., 

2004) 


As  part  of  U.S.  Southern  Command’s  (USSOUTHCOM)  regional  engagement 
strategy,  the  U.S.  military  executes  HCA  projects  in  selected  countries  within  its  area  of 
responsibility.  Operation  NEW  HORIZONS  2014  was  executed  in  Belize  from  April  to 
June  2014  with  stated  objectives  to  exercise  USSOUTHCOM’s  ability  to  plan  and 
execute  a  deployment  of  personnel  and  equipment,  as  well  as  build  partnerships  with  host 
nation  military  personnel.  Among  the  activities  executed  included  the  construction  of 
four  schools  and  one  clinic  in  three  communities.  These  projects  ranged  in  costs  from 
$75  thousand  to  $204  thousand  and  were  executed  by  the  820th  RED  HORSE  Squadron 
in  concert  with  US  Army,  US  Marine  Corps,  and  Belize  Defense  Force  (BDF)  engineer 
units.  While  none  of  the  exercise  objectives  were  to  build  the  legitimacy  of  the 
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government,  the  exercise  presented  an  opportunity  to  measure  the  impact  of  a 
humanitarian  project  on  a  population’s  satisfaction  with  government  services. 

Belize  is  a  former  British  colony  in  Central  America,  with  a  population  of  340 
thousand  people  and  an  annual  Gross  Domestic  Product  of  $1.6  billion  (CIA:  The  World 
Factbook,  2014).  The  economy  is  primarily  based  on  tourism  and  agriculture  with 
limited  industrial  capacity  (World  Travel  &  Tourism  Council,  2012).  Overall,  Belize  has 
the  3rd  lowest  satisfaction  with  municipal  services  rating  in  Latin  America,  with  less  than 
20%  of  citizens  rating  services  favorable  (Montalvo,  2009;  LAPOP,  2014).  Considering 
this  baseline  information,  it  would  be  expected  that  on  average  government  services  are 
in  need  of  improvement  and  that  improvements  may  result  in  higher  ratings  of  citizen 
satisfaction. 

Methodology 

Using  Van  Ryzin’s  model  and  survey  instruments  as  a  baseline  (Van  Ryzin  2004; 
Van  Ryzin  2005),  a  survey  was  developed  to  test  the  impact  of  a  humanitarian  project  on 
citizen  satisfaction  using  a  1-5  Likert  scale.  The  results  of  the  survey  were  analyzed  by 
Hansen  (2015),  who  confirmed  that  effects,  except  the  effect  of  expectations  on 
disconfirmation,  were  statistically  significant.  The  conclusion  of  his  research  is  that  the 
model  shown  in  Figure  13  is  a  valid  model  for  assessing  citizen  satisfaction  with  all 
effects  confirmed  individually.  His  model  thus  formed  the  basis  for  this  research  effort, 
which  also  used  the  same  survey  data  set  for  additional  analysis. 
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Figure  13:  Citizen  Satisfaction  Model  (Hansen,  2015) 

Resampling  is  a  method  used  to  increase  the  sample  size  of  a  limited  data  set.  By 
randomly  selecting  a  set  of  values  from  a  given  data  set  and  placing  these  values  into  a 
new  data  set,  additional  samples  can  be  generated.  Each  set  of  values  in  the  original  data 
has  an  equal  chance  of  being  selected,  and  every  subsequent  resample  may  select  an 
already  sampled  set  of  data.  Using  this  method,  it  is  expected  that  the  values  of  the  larger 
resampled  data  set  will  represent  the  true  values  of  the  mean  and  standard  deviation  of 
the  population. 

Linear  regression  was  used  to  determine  the  relationships  between  the  selected 
predictor  variables  and  overall  citizen  satisfaction.  Regression  models  were  analyzed 
using  JMP  ®  1 1  Pro  produced  by  SAS  ®  using  the  same  data  collected  and  analyzed  by 
Hansen  (2015).  Regression  models  included  single-factor,  multiple  factor,  and 
interaction  effects  that  were  determined  to  be  significant.  These  models  were  analyzed 
for  percent  of  variance  explained  (R  )  as  well  as  the  coefficients  of  regression  (jl)  for 
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each  variable  of  interest.  These  regression  models  formed  the  baseline  of  further  analysis 
of  the  strength  of  predictor  variables  and  in  developing  a  simulation  of  the  response 
variable,  which  was  defined  as  citizen  satisfaction.  Regression  analysis  was  conducted 
on  both  the  baseline  survey  data  collected  as  well  as  the  resampled  data  using  the 
resampling  methodology  previously  discussed. 

Dominance  analysis  is  a  method  used  to  determine  the  importance  of  predictor 
variables  on  the  overall  regression  relationship.  While  regression  coefficients  can 
indicate  the  strength  of  the  predictor  on  the  outcome,  this  analysis  can  be  problematic 
when  predictors  are  correlated  or  even  collinear  (Tonidandel  &  LeBreton,  201 1).  A 
method  introduced  by  Budescu  in  1993  analyzes  the  change  in  the  R“  of  a  model  by 
incrementally  including  each  predictor  into  the  model  and  ranks  the  predictors  in 
importance  (Budescu,  1993).  Using  this  core  method,  Dominance  Analysis  has  been 
refined  and  a  method  of  pair-wise  comparisons  of  the  influence  of  each  variable  is 
conducted  in  order  to  determine  which  predictor  has  the  greatest  effect  on  explaining 
variance  in  the  model  (Azen  &  Budescu,  2003).  This  method  was  selected  for  use  in  this 
analysis  due  to  the  nature  of  the  correlations  of  the  predictor  variables. 

Additive  regression  modeling  was  used  to  explore  the  relationship  and  effects  of 
the  predictor  variables  on  citizen  satisfaction.  The  results  of  the  regression  analysis 
provided  regression  coefficients  and  an  intercept  which  were  used  to  build  a  prediction 
model  for  citizen  satisfaction  based  on  the  most  dominant  predictors.  Using  this  formula, 
various  “scenarios”  were  developed  based  on  the  possible  situations  that  could  be 
encountered  based  on  possible  answers  on  a  1-5  Likert  scale.  The  scenario  analysis  used 
these  values  of  the  selected  predictors  to  determine  a  value  for  citizen  satisfaction. 
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Results 


Resampling  analysis  was  conducted  on  the  survey  data.  Table  3  shows  that  the 
summary  descriptive  statistics  of  the  resampling  data  set  were  almost  identical  to  the 
actual  data  and,  as  expected,  did  not  yield  significantly  different  results.  Similar  analysis 
was  conducted  on  the  data  attempting  to  decompose  the  data  into  smaller  subsets,  but  the 
results  for  all  cases  were  identical  to  the  original  sample.  These  additional  tables  are 
included  in  Appendix  D. 


Table  3:  Comparison  of  Original  and  Resampled  Descriptive  Statistics 


Original  Data,  n  =  627 

Resampled  Data,  n  =  1( 

),000 

Variable 

E 

P 

D 

Sat 

E 

P 

D 

Sat 

Mean 

4.50 

2.96 

2.87 

2.58 

4.50 

2.95 

2.87 

2.57 

Median 

4.67 

2.88 

3.00 

2.33 

4.67 

2.88 

3.00 

2.33 

Std  Dev 

0.62 

0.75 

0.93 

0.96 

0.62 

0.74 

0.93 

0.96 

E  =  Expectation,  P  =  Performance,  D  =  Disconfirmation,  Sat  =  Citizen  Satisfaction 


Building  on  the  results  from  Hansen  (2015),  all  variables  were  confirmed  to  be 
significant  predictors  of  citizen  satisfaction  individually  and  were  tested  in  an  additive 
regression  model.  These  results  are  listed  in  Table  4.  While  the  initial  research  into  these 
relationships  indicated  that  all  variables  were  significant,  not  all  variables  remained 
significant  when  all  variables  were  considered  in  a  single  additive  regression  model.  The 
predictor  of  expectations  became  insignificant  at  the  p  =  0.05  level,  and  the  intercept  of 
the  regression  equation  was  also  insignificant.  Of  note  is  that  the  mean  of  citizen 
satisfaction  from  the  Hansen  survey  data  was  2.576,  which  is  similar  to  the  results  of  the 
LAPOP  data  (2014). 
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Table  4:  Descriptive  Statistics  of  Hansen  Model,  R2  =  0.565 


Predictor 

Mean 

GO 

Standard 

Deviation 

Regression 

Coefficient 

(P) 

p-Value 

Intercept 

- 

- 

0.032 

0.8798 

Expectation 

4.503 

0.622 

-0.061 

0.1348 

Performance 

2.956 

0.745 

0.566 

<0.0001 

Disconfirmation 

2.866 

0.929 

0.380 

<0.0001 

Centered 
Performance  - 
Disconfirmation 
Interaction 

0.437 

0.797 

0.139 

<0.0001 

Satisfaction 

2.576 

0.957 

- 

- 

Upon  inspection  of  the  regression  coefficients  listed  in  Table  4,  it  would  appear 
that  performance  is  the  largest  influencer  of  citizen  satisfaction,  with  disconfirmation 
being  the  second  largest  influencer.  However,  this  can  be  misleading  as  analysis  based 
only  upon  inspection  of  regression  coefficients  may  be  inaccurate  when  variables  are 
correlated.  The  correlation  between  all  variables  is  presented  in  Table  5,  which  illustrates 
correlation  coefficients  greater  than  0.6  between  citizen  satisfaction  and  both 
performance  and  disconfirmation,  as  well  as  between  performance  and  disconfirmation. 
Due  to  the  nature  of  the  correlations  between  predictor  variables,  it  is  appropriate  to 
analyze  these  predictors  through  a  dominance  analysis. 


Table  5:  Correlation  Table  of  Variables 


Expectation 

Performance 

Disconfirmation 

Satisfaction 

Expectation 

1.000 

- 

- 

- 

Performance 

-0.022 

1.000 

- 

- 

Disconfirmation 

-0.069 

0.632* 

1.000 

- 

Satisfaction 

-0.064 

0.684* 

0.658* 

1.000 

*  Correlation  is  significant  at  p  <  0.< 

3001 
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The  citizen  satisfaction  data  were  analyzed  using  the  dominance  analysis  method 
pioneered  by  Budescu  (1993),  with  the  results  being  presented  in  Table  6.  The 
dominance  of  the  predictors  can  be  determined  from  the  differences  in  R"  of  each  model. 
In  each  of  the  models  where  a  single  predictor  is  used,  the  R”  of  that  value  accounts  for 
the  contribution  of  that  predictor,  which  is  indicated  in  the  first  row  of  Table  6  in  the  row 
titled  “k  =  0  Average.”  Similarly,  for  each  row  in  the  table,  the  contribution  of  the 
predictor  is  calculated  by  taking  the  value  of  the  R"  from  the  model  where  each  predictor 
of  that  row  plus  the  predictor  of  that  column  is  included,  and  then  the  value  of  the  R“  for 
the  model  of  that  row  is  subtracted  from  it.  For  each  category  of  k,  the  contribution  from 
each  predictor  is  averaged.  The  final  row  shows  all  of  the  predictors  included  in  the 
model,  thus  the  cells  for  each  predictor  in  that  row  are  empty.  The  General  Dominance 
row  represents  the  average  of  all  of  the  k  averages  for  that  predictor.  To  calculate  the 
Rescaled  Dominance,  the  General  Dominance  of  that  predictor  is  divided  by  the  sum  of 
the  General  Dominance  values  for  all  predictors  and  multiplied  by  100%  to  represent  the 
percentage  of  R"  change  that  the  predictor  contributed  to  the  full  model. 

As  presented  in  Table  6,  performance  is  the  largest  contributor  to  R-  in  the  models 
tested,  with  disconfirmation  also  being  close.  These  two  predictors  together  account  for 
94%  of  the  change  in  R  in  the  models,  but  the  interaction  of  performance  and 
disconfirmation  only  accounts  for  5.38%  of  the  change  in  R“.  Furthermore,  the 
contribution  of  expectation  to  these  models  is  minimal,  with  one  instance  of  expectation 
decreasing  R"  when  added  to  the  model  with  disconfirmation.  While  the  best  R”  of  these 
tested  models  was  0.565  with  all  four  predictors  included,  the  additional  contribution  of 
expectation  only  increased  the  model  R“  by  0.001. 
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Table  6:  Dominance  Analysis  of  Predictors  of  Citizen  Satisfaction 


Model 

Model  R2 

Ad 

Iditional  Contribution  of: 

A, 

a2 

A? 

A, 

k  =  0  Average 

0.002 

0.467 

0.433 

0.033 

A; 

0.002 

0.466 

0.430 

0.036 

X2 

0.467 

0.002 

0.085 

0.016 

X3 

0.433 

-0.001 

0.119 

0.015 

A, 

0.033 

0.005 

0.450 

0.414 

k-  1  Average 

0.002 

0.467 

0.433 

0.033 

A/A2 

0.468 

0.083 

0.018 

A/A; 

0.432 

0.120 

0.132 

A/A, 

0.039 

0.448 

0.409 

a2  A, 

0.552 

0.000 

0.013 

x2x4 

0.483 

0.003 

0.081 

A, A, 

0.448 

0.000 

0.117 

k-2  Average 

0.001 

0.228 

0.191 

0.054 

A/A2A, 

0.552 

0.013 

X,X2X4 

0.486 

0.079 

A/A/A, 

0.448 

0.117 

x2x4x4 

0.564 

0.001 

k  =  3  Average 

0.001 

0.117 

0.079 

0.013 

x,x2x3x4 

0.565 

General 

Dominance 

0.002 

0.289 

0.253 

0.031 

Rescaled 

Dominance 

0.275% 

50.3% 

44.0% 

5.38% 

X]  =  Expectation,  A2  =  Performance,  A?  =  Disconfirmation,  X4  =  Centered  Interaction 
of  Performance  and  Disconfirmation,  and  k  =  Number  of  Predictor  Variables  in 
Model 


With  the  conclusion  that  performance,  disconfirmation,  and  the  interaction  of 
performance  and  disconfirmation  contribute  the  most  to  explaining  variance  in  citizen 
satisfaction,  a  new  model  was  developed  and  tested.  This  new  model  did  not  include 
expectation  as  a  predictor  variable.  The  descriptive  statistics  for  this  model  are  included 
in  Table  7.  While  the  values  for  the  regression  coefficients  did  not  change  substantially 
for  the  predictor  variables,  the  intercept  value  changed  and  became  statistically 
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significant  at  the  p  =  0.05  level.  This  model  is  therefore  more  statistically  accurate, 
although  the  R“  of  the  model  is  still  only  0.564,  and  returning  a  statistically-significant 
value  for  an  intercept  enabled  a  more  accurate  prediction  model  for  citizen  satisfaction. 
Using  these  values,  an  additive  regression  prediction  formula  was  built  using  Equation  1, 
where  Y  =  citizen  satisfaction,  pn  =  regression  coefficient,  Xn  =  value  of  prediction 
variable,  and  pn  =  mean  of  prediction  variable. 


Table  7:  Descriptive  Statistics  of  New  Model,  R2  =  0.564 


Predictor 

Variable 

Mean 

00 

Standard 

Deviation 

Regression 

Coefficient 

(P) 

p-Value 

Intercept 

- 

- 

- 

-0.248 

0.0187 

Performance 

X, 

2.956 

0.745 

0.566 

<0.0001 

Disconfirmation 

X2 

2.866 

0.929 

0.380 

<0.0001 

Centered 
Performance  - 
Disconfirmation 
Interaction 

X3 

0.437 

0.797 

0.139 

<0.0001 

Y  =  J30  +  J31X1+/32X2+&(X1  -//,)( X2  -//2)  (1) 

Using  Equation  1,  the  predicted  values  of  citizen  satisfaction  were  calculated  for 
each  set  of  actual  values  for  performance  and  disconfirmation.  The  predicted  values  of 
citizen  satisfaction  plotted  versus  the  actual  survey  values  are  presented  in  Figure  14. 
Since  the  measure  of  actual  citizen  satisfaction  was  an  average  of  three  questions  on  a  1-5 
Likert  scale,  these  data  points  cluster  in  increments  of  0.33  along  the  x-axis.  In  some 
cases,  a  respondent  did  not  answer  all  three  satisfaction  measures,  which  resulted  in  an 
average  of  two  values  and  thus  returned  some  values  in  increments  of  0.5.  In  two  cases, 
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the  predicted  value  of  satisfaction  exceeded  the  maximum  possible  value  of  5  on  the 
survey  instrument.  If  the  prediction  model  were  a  perfect  fit  with  a  R~  of  1.0,  the  data 
points  would  form  a  straight  line  upwards  from  the  origin  at  a  45 -degree  angle;  however, 
this  model  only  had  an  R  of  0.564,  which  resulted  in  significant  variation  in  the 
predicted  data  points. 

To  further  explore  the  relationships  of  performance  and  disconfirmation  on  citizen 
satisfaction,  Equation  1  was  used  to  develop  a  scenario  based  on  values  of  performance 
and  expectation.  The  prediction  formula  was  used  to  determine  the  predicted  value  of 
citizen  satisfaction  for  both  values  of  predictor  variables.  This  analysis  is  presented  in 
Table  8,  with  each  possible  value  of  disconfirmation  represented  by  column,  each 
possible  value  of  performance  by  row,  and  the  expected  value  of  citizen  satisfaction  for 
each  column  and  row  combination  of  predictors  in  the  individual  cells.  The  results  show 
that  the  rating  of  citizen  satisfaction  increases  much  faster  as  values  of  performance 
increase  versus  the  values  of  disconfirmation.  For  example,  citizen  satisfaction  only 
increases  by  1.54  as  disconfirmation  increases  from  1  to  5  when  performance  stays  at  3, 
whereas  satisfaction  increases  by  2.34  as  performance  increases  from  1  to  5  and 
disconfirmation  remains  at  3.  The  value  of  3  is  closest  to  the  mean  of  both  predictor 
variables,  which  results  in  the  interaction  of  the  predictors  having  the  smallest  effect  on 
satisfaction. 

Overall,  these  results  present  the  possibility  that  citizen  satisfaction  may  be 
predicted  given  performance  and  disconfirmation,  but  the  results  are  vulnerable  to  error. 
The  R"  of  the  model  was  only  0.564,  which  resulted  in  wide  variation  in  the  predicted 
results  compared  to  the  actual  values  of  citizen  satisfaction. 
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Figure  14:  Predicted  vs.  Actual  Citizen  Satisfaction 


Table  8:  Scenario  Analysis  of  Citizen  Satisfaction 


Disconfirmation  Value 

1 

2 

3 

4 

5 

Performance 

Value 

1 

1.21 

1.31 

1.42 

1.53 

1.64 

2 

1.51 

1.76 

2.01 

2.25 

2.50 

3 

1.82 

2.21 

2.59 

2.98 

3.36 

4 

2.13 

2.65 

3.18 

3.70 

4.23 

5 

2.43 

3.10 

3.76 

4.43 

5.00 
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Discussion 


The  methodology  and  results  of  this  analysis  partially  confirm  the  models 
developed  by  Van  Ryzin  (2004)  and  tested  by  Hansen  (2015);  however,  the  effect  of 
expectations  on  both  disconfirmation  and  citizen  satisfaction  was  not  supported.  Using 
the  revised  model,  it  is  possible  to  predict  citizen  satisfaction  given  performance  and 
disconfirmation  data;  however,  this  prediction  comes  with  a  wide  prediction  interval  of 
approximately  ±1.25,  which  makes  it  difficult  to  accurately  predict  satisfaction  using  a  1- 
5  Likert  scale. 

The  results  of  this  research  can  be  improved  with  some  changes  survey  execution 
methodology.  Survey  data  collection  was  originally  intended  to  be  conducted  using  BDF 
personnel  interviewing  citizens  of  Belize,  with  BDF  personnel  recording  the  answers  to 
surveys  and  the  location  of  collection.  The  situation  during  the  early  portion  of  the 
exercise  necessitated  these  surveys  be  distributed  in  print  to  BDF  personnel  to  distribute 
to  their  home  towns  prior  to  an  extended  leave  period  in  conjunction  with  Easter;  in  turn, 
many  BDF  personnel  handed  out  paper  copies  of  the  survey  for  citizens  to  fill  out, 
resulting  in  missing  data,  multiple  Likert  scale  selections  per  measure,  and  various  other 
data  irregularities.  This  approach  was  also  problematic  considering  that  although  the 
official  language  in  Belize  is  English,  only  3.9%  of  the  population  listed  English  as  their 
language  in  the  2000  census,  with  the  largest  percentages  of  language  being  Spanish  at 
46%  and  Creole  at  33%.  Additionally,  the  country’s  adult  literacy  rate  is  77%  (CIA:  The 
World  Factbook,  2014).  These  factors  may  bring  into  question  the  validity  of  the  survey 
data  and  by  extension  the  overall  results. 
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Another  improvement  to  survey  methodology  is  in  the  collection  of  surveys  with 
respect  to  the  areas  of  interest.  In  the  initial  round  of  surveys,  only  56  of  172  useable 
surveys  were  collected  from  locations  where  a  project  was  executed;  in  the  following 
round,  247  of  455  useable  surveys  were  collected  from  two  of  the  three  locations  where  a 
project  was  executed  (Hansen,  2015).  The  lack  of  a  sufficient  number  of  surveys  from 
the  initial  round  limited  the  ability  to  compare  the  results  before  and  after  a  project  was 
executed.  The  survey  collection  also  combined  all  non-project  location  survey  data  into  a 
control  group  for  analysis;  however,  this  resulted  in  wide  variation  in  the  data  that  could 
be  explained  by  location- specific  issues,  and  some  locations  were  not  represented  in  both 
survey  rounds.  Future  survey  research  should  attempt  to  collect  survey  data  in  sufficient 
numbers  from  project  locations,  and  should  designate  a  specific  control  location  for 
sampling. 

The  timing  of  survey  collection  was  also  problematic.  Four  of  the  projects 
executed  were  schools  and  surveys  were  collected  initially  in  April  and  again  in  June. 
However,  the  citizens  in  the  area  around  these  schools  are  unlikely  to  have  noticed  any 
substantial  change  in  government  services  as  the  children  that  normally  attend  these 
schools  were  on  summer  vacation  during  the  June  survey  period.  A  citizen  would  likely 
not  have  an  informed  judgment  on  quality  of  school  service  until  after  schools  reopened 
sometime  in  the  fall  of  2015.  In  the  meantime,  it  is  possible  that  the  noise,  dust,  and 
truck  traffic  that  accompanied  construction  of  the  schools  may  have  irritated  local 
citizens,  resulting  in  biased  survey  data.  Data  fidelity  can  be  improved  by  sampling 
citizens  in  the  area  around  these  project  locations  at  a  later  time  after  construction  and 
after  school  has  returned  to  session. 
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Aside  from  survey  execution  issues,  it  is  possible  that  citizen  satisfaction  may  be 
influenced  by  another  unmeasured  variable.  Expectation  was  shown  to  be  an 
insignificant  predictor,  but  the  model  only  explained  56%  of  the  variation  in  citizen 
satisfaction.  Data  from  previous  research  indicates  that  Belize  has  a  low  rating  of  trust  in 
government  compared  to  other  Latin  American  countries  (LAPOP,  2014).  This  may 
indicate  that  trust  in  government,  as  well  as  other  factors  such  as  perceived  justice,  may 
influence  a  citizen’s  satisfaction  judgment  with  government  services. 

Conclusion 

This  research  presents  a  method  for  predicting  citizen  satisfaction.  While  the 
overall  accuracy  of  the  results  leave  room  for  improvement,  it  is  possible  that 
improvements  in  survey  deployment  and  collection  timelines  may  improve  these  results. 
This  method  may  be  useful  in  a  stability  or  COIN  operations  for  determining  a 
population’s  baseline  level  of  citizen  satisfaction  with  government  services.  This 
information  may  help  guide  selection  of  humanitarian  projects  to  address  the  needs  of  a 
population  and  improve  their  rating  of  satisfaction  with  their  government.  Additionally, 
this  model  and  method  may  be  useful  in  municipalities  within  the  U.S.  in  selecting  where 
to  invest  in  public  infrastructure. 
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III.  Results  and  Conclusion 


Research  Questions  Revisited 

This  research  posed  the  problem  of  how  to  measure  the  effectiveness  of  a 
humanitarian  project.  The  research  applied  citizen  satisfaction  models  and  statistical 
analysis  techniques  to  evaluate  a  method  for  assessing  citizen  satisfaction  in  conjunction 
with  a  humanitarian  assistance  project.  Results  have  been  processed  and  the  research 
questions  can  now  be  explicitly  addressed: 

Can  satisfaction  be  predicted  when  the  expectations  of  the  population  is  known? 

Initial  results  from  Van  Ryzin  (2004,  2005)  and  Hansen  (2015)  indicated  that 
expectation  is  a  weak  predictor  of  citizen  satisfaction.  Through  regression  and 
dominance  analysis,  it  appears  that  citizen  satisfaction  cannot  be  predicted  based  on 
known  expectations.  Furthermore,  expectation  had  an  insignificant  effect  when 
combined  in  a  model  with  performance  and  disconfirmation. 

Can  satisfaction  be  predicted  when  the  population ’s  rating  of  government  service 
performance  is  known? 

Results  from  previous  research  suggest  that  performance  is  a  strong  predictor  of 
citizen  satisfaction  (Van  Ryzin,  2005).  When  performance  is  selected  as  the  only 
predictor  of  citizen  satisfaction,  the  analysis  results  in  an  R"  of  0.467;  the  model  is 
responsible  for  50.3%  of  the  explained  variance  in  regression  analysis  with  additional 
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predictors.  While  performance  is  the  strongest  predictor  of  citizen  satisfaction,  it  should 
be  further  combined  with  other  factors  to  improve  prediction  accuracy. 

Can  satisfaction  be  predicted  when  the  difference  between  the  population ’s 
expectations  and  perception  of  government  services  performance  is  known? 

Results  from  previous  research  suggest  that  the  difference  between  the 
population’s  expectations  and  perception  of  performance,  also  termed  disconfirmation,  is 
the  strongest  predictor  of  citizen  satisfaction  (Van  Ryzin,  2005).  The  result  from  this 
analysis  suggest  that  disconfirmation,  while  a  strong  predictor  with  an  R~  of  0.433  when 
considered  alone,  is  not  as  strong  as  performance  but  still  accounted  for  44.0%  of  the 
explained  variance  in  regression  analysis.  These  results  suggest  that  disconfirmation  is  a 
good  predictor,  but  it  should  also  be  considered  with  other  factors  to  improve  prediction 
accuracy. 

Which  factors  are  most  influential  on  citizen  satisfaction? 

According  to  this  analysis,  the  most  influential  factors  on  citizen  satisfaction  are 
performance  and  disconfirmation.  This  conclusion  is  also  supported  by  previous 
research,  but  the  magnitudes  of  these  relationships  are  slightly  different.  In  previous 
research,  disconfirmation  was  shown  to  be  the  most  influential,  but  these  data  support 
performance  as  the  strongest  predictor. 
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Can  guidance  be  given  in  target  selection  to  increase  return  on  investment  on 
infrastructure  targets  with  respect  to  citizen  satisfaction? 

To  maximize  the  effect  of  humanitarian  assistance,  targets  should  be  selected  in 
areas  where  citizens  have  low  ratings  of  disconfirmation  and  performance  of  government 
services.  This  condition  may  represent  a  situation  in  which  there  is  a  potential  to  improve 
these  ratings  through  investment  in  low-performing  services  that  may  result  in  higher 
gains  in  citizen  satisfaction  when  compared  to  the  costs  or  time  necessary  to  improve 
quality  of  services  that  are  rated  as  high  performance. 

Is  the  model  of  predicting  citizen  satisfaction  sensitive  to  sample  size? 

The  data  analysis  conducted  through  resampling  returned  statistically  identical 
results  as  expected.  As  a  result,  the  outputs  of  regression  analysis  were  identical  to  the 
original  data  set.  The  model  was  not  demonstrated  to  be  sensitive  to  sample  size  given 
the  suggested  minimum  samples  for  data  analysis  are  met.  This  anticipated  minimum 
sample  size  was  76  for  each  data  point  before  and  after  the  treatment  (Cohen,  1992). 

Review  of  Findings 

Overall,  this  model  was  supported  by  the  data,  but  there  are  improvements  that 
can  be  made.  Improvements  in  data  collection  may  improve  results,  but  further  analysis 
of  the  model  is  warranted.  The  distribution  of  the  model  output  of  predicted  citizen 
satisfaction  appears  similar  to  the  actual  data  with  some  discrepancies.  Figure  15  shows 
the  distribution  of  the  actual  survey  data,  and  Figure  16  shows  the  distribution  of  the 
predicted  value  of  citizen  satisfaction  given  performance  and  disconfirmation  values. 
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Upon  inspection,  it  appears  that  the  model  may  return  insufficient  results  in  the  extremes 
of  citizen  satisfaction.  The  actual  data  in  Figure  15  show,  a  large  number  of  surveys  that 
rate  in  the  1,  1.33,  and  1.67  range,  but  the  predicted  data  in  Figure  16  show  few  results 
below  1.33  but  many  more  below  2;  similarly,  the  actual  data  show  many  more  results 
above  4,  whereas  the  predicted  values  begin  to  taper  off  prior  to  values  of  4.  This  is 
expected  considering  the  regression  analysis  attempted  to  fit  a  normal  curve  to  the  data 
which  inherently  reduces  the  data  points  in  the  “tails”  of  the  distributions. 


Actual  Citizen  Satisfaction 

Figure  15:  Histogram  of  Actual  Citizen  Satisfaction 
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Predicted  Citizen  Satisfaction 

Figure  16:  Histogram  of  Predicted  Citizen  Satisfaction 

Further  analysis  was  conducted  on  these  values.  The  difference  between  the 
predicted  and  actual  values  was  calculated  for  each  record  in  the  data.  This  difference 
was  then  divided  by  the  actual  value  to  return  a  percent  error;  a  predicted  value  higher 
than  the  actual  value  is  indicated  by  a  negative  percent  error  value,  and  a  predicted  value 
higher  than  the  actual  value  is  indicated  by  a  negative  percent  error  value.  These  results 
are  plotted  in  a  histogram  in  Figure  17.  This  figure  indicates  that  the  model  is  under¬ 
estimating  the  value  of  satisfaction  frequently  by  a  small  margin,  but  also  greatly 
overestimates  the  rating  of  satisfaction  sporadically  out  to  an  extreme  of  154%.  The  long 
tail  in  the  data  suggests  that  further  analysis  of  these  extreme  values  may  yield  further 
explanation. 
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Predicted  vs  Actual  Percent  Error 

Figure  17:  Histogram  of  Predicted  vs  Actual  Percent  Error 

The  previously  discussed  distributions  also  returned  descriptive  statistics,  which 
are  included  in  Table  9.  These  statistics  indicate  that  the  predicted  values  for  citizen 
satisfaction  have  a  tighter  distribution  around  the  mean  with  a  lower  standard  deviation, 
where  the  actual  vales  have  a  flatter,  less  normal  distribution.  While  both  distributions 
have  identical  means,  the  predicted  value  distribution  has  a  higher  median  value. 
Additionally,  the  distribution  of  error  indicates  that  predicted  citizen  satisfaction  is  on 
average  7.76%  higher  than  the  actual  value. 
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Table  9:  Descriptive  Statistics  of  Citizen  Satisfaction  Distribution 


Distribution 

Mean 

00 

Median 

Standard 

Deviation 

(<0 

Upper 
95%  Mean 

Lower 
95%  Mean 

Actual 

Satisfaction 

2.58 

2.33 

0.96 

2.65 

2.50 

Predicted 

Satisfaction 

2.58 

2.48 

0.72 

2.63 

2.51 

Percent 

Error 

-7.76% 

0.07% 

32.2% 

-5.24% 

-10.3% 

A  plot  of  the  predicted  values  and  prediction  interval  compared  to  the  actual 
satisfaction  values  is  presented  in  Figure  18.  The  vertical  axis  represents  citizen 
satisfaction  and  the  x  axis  indicates  the  row  of  data.  The  upper  red  and  lower  blue  lines 
represent  the  95%  confidence  interval  of  the  predicted  value,  and  the  green  stars  indicate 
the  actual  value  of  citizen  satisfaction.  The  values  of  predicted  citizen  satisfaction  were 
listed  along  the  x-axis  from  lowest  to  highest,  with  the  predicted  value  of  citizen 
satisfaction  in  orange  between  the  upper  red  and  lower  blue  lines.  The  actual  values  of 
citizen  satisfaction  were  an  average  of  three  questions  in  the  survey  instrument,  which 
resulted  in  the  actual  values  clustering  in  0.333  increments,  with  a  few  instances  of  a 
value  that  was  averaged  from  two  values  due  to  a  missing  response.  The  degree  of 
variation  between  the  predicted  and  actual  values  does  not  appear  to  have  a  clear 
relationship  or  pattern.  The  scale  for  responses  in  the  survey  was  a  1-5  Likert  scale; 
therefore,  the  actual  values  all  range  between  one  and  five,  while  the  predicted  values  and 
confidence  intervals  range  well  outside  both  of  these  extremes. 
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Figure  18:  Predicted  and  Actual  Citizen  Satisfaction  Values 


The  additional  figures  presented  in  this  section  yield  interesting  results.  The 
histograms  and  plot  graphically  show  the  high  degree  of  unexplained  variation  in  the 
model.  This  may  indicate  that  there  is  an  unmeasured  variable  influencing  the  value  of 
citizen  satisfaction  and  merits  further  research. 
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Significance  of  Research 

This  research  contributed  to  the  body  of  knowledge  regarding  the  relationship  of 
expectation,  perception  of  performance,  and  disconfirmation  on  citizen  satisfaction  with 
government  services  in  a  less  industrialized  country  that  may  be  applied  towards  a  target 
selection  methodology  for  humanitarian  assistance  projects.  This  research  may  also 
benefit  municipal  governments  by  contributing  to  the  body  of  knowledge  by  further 
exploring  the  influence  of  government  service  performance  on  citizen  satisfaction. 

Future  Research 

Problems  encountered  with  the  survey  methodology  were  discussed  in  Chapter  II, 
but  further  explanation  is  necessary.  Follow-on  survey  analysis  of  these  project  locations 
in  Belize  is  warranted  to  determine  the  true  impact  of  these  projects  after  citizens  have 
opportunities  to  evaluate  the  improvement  of  these  government  services,  and  a  more 
robust  sampling  plan  should  be  developed  and  implemented  regarding  the  sampling  of 
both  control  and  treatment  locations.  Future  survey  data  collection  should  be  done 
verbally  with  data  recorded  by  trained  personnel.  Additionally,  the  survey  instrument 
should  be  translated  into  Spanish  for  another  survey  deployment,  and  should  also  be 
back-translated  into  English  to  confirm  that  the  initial  translation  is  accurate. 

It  is  unclear  whether  variation  in  the  data  can  be  improved  only  by  improvement 
in  survey  methodology,  or  if  there  is  an  unmeasured  variable  that  is  also  influencing 
citizen  satisfaction.  Previous  research  by  LAPOP  have  surveyed  trust  in  government, 
which  indicated  that  citizens  in  Belize  have  a  relatively  low  rating  of  their  government 
compared  to  other  Latin  American  countries.  Further  research  should  include  a  series  of 
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measures  to  survey  a  citizen’s  trust  in  government,  as  well  as  variables  such  as  their 
perceptions  of  fairness  and  justice,  to  determine  if  these  have  an  effect  on  a  citizen’s 
satisfaction  with  government  services. 

Determining  the  impact  of  foreign  aid  has  been  illustrated  as  difficult  task  in  need 
of  further  research.  In  particular,  the  relationship  of  distance  from  a  given  project  is  not 
well  understood  (Fischerkeller,  2011).  This  relationship  can  be  further  explored  using  a 
combination  of  surveys  and  GIS  software  if  the  project  location  and  attributes  are  known 
along  with  the  location  of  the  survey  collection  point  or  the  respondent’s  residence 
location.  Using  these  data  sets,  a  relationship  between  distance,  or  areal  units,  to  citizen 
satisfaction  may  be  developed.  For  example,  survey  collected  regarding  a  school  project 
from  respondents  that  live  within  that  school  district  compared  to  respondents  outside  the 
school  district  may  have  different  results.  Does  a  citizen  that  lives  within  that  school 
district  reflect  a  higher  level  of  satisfaction  after  a  project  is  accomplished?  Will  a  citizen 
that  lives  outside  that  school  district  see  an  increase  in  satisfaction  due  to  this 
improvement,  or  possibly  see  a  decrease  in  satisfaction  due  to  an  improvement  that  did 
not  benefit  him  or  her?  These  are  important  questions  that  could  improve  target  selection 
for  humanitarian  projects. 

An  additional  impact  of  foreign  aid  that  is  unknown  is  at  what  rate  the  impact  of 
aid  diminishes  in  the  collective  memory  of  the  population.  The  relationship  of  decay  of 
an  impact  with  time  is  not  well  understood  and  is  often  cited  as  an  important  question  for 
determining  targeting  and  execution  of  further  aid  (Fischerkeller,  2011).  This 
information  could  be  used  to  maximize  the  sustainment  of  an  increased  level  of 
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satisfaction  with  an  optimization  of  resources  expended  over  the  course  of  several 
projects  in  series  for  a  long-term  impact. 

Additional  survey  sampling  of  the  areas  where  projects  were  executed  in 
Operation  NEW  HORIZONS  14  in  the  following  years  can  assist  in  answering  both  of 
the  previously  identified  questions.  Using  the  survey  data  collected  in  this  research  and 
comparing  to  follow-on  results  with  higher  resolution  location  information  can  build 
upon  the  body  of  knowledge  in  this  area. 

Summary 

This  research  explored  the  relationships  of  expectation,  performance,  and 
disconfirmation  on  citizen  satisfaction.  This  research  built  upon  the  analysis  of  Hansen 
(2015)  and  used  the  same  survey  data  set  for  regression  and  dominance  analysis  which 
resulted  in  the  development  of  a  new  model  for  predicting  citizen  satisfaction.  This 
research  proposed  a  model  that  may  be  used  to  predict  citizen  satisfaction  given 
performance  and  disconfirmation,  which  may  be  useful  in  target  selection  for 
humanitarian  assistance  projects.  This  research  was  not  conclusive  in  the  determination 
of  citizen  satisfaction,  but  it  contributes  to  the  body  of  knowledge  with  respect  to  citizen 
satisfaction  decisions. 
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Appendix  A:  Survey  Instrument 


Section  1: 

What  do  you  believe  are  the  three  most  important  government 
services?  Rank  them  from  1  to  3  where  1  is  most  important. 

1.1)1 

1.2)2 

1.3)3 

Section  2: 

On  a  scale  from  1  to  5,  with  1  being  strongly  disagree  and  5 
being  strongly  agree,  how  would  you  rate  each  of  the  following 
statements  for  your  community? 

Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Strongly 

Agree 

2.1)  I  expect  the  government  to  provide  quality  education 

1 

2 

3 

4 

5 

2.2)  I  expect  the  government  to  provide  quality  health  care 

1 

2 

3 

4 

5 

2.3)  I  expect  the  government  to  provide  other  public  services 

1 

2 

3 

4 

5 

2.4)  I  am  happy  with  the  schools 

1 

2 

3 

4 

5 

2.5)  I  am  happy  with  school  management 

1 

2 

3 

4 

5 

2.6)  I  am  happy  with  the  health  care 

1 

2 

3 

4 

5 

2.7)  I  am  happy  with  the  local  police  service 

1 

2 

3 

4 

5 

2.8)  I  am  happy  with  the  garbage  removal 

1 

2 

3 

4 

5 

2.9)  I  am  happy  with  the  quality  of  drinking  water 

1 

2 

3 

4 

5 

2.10)  I  am  happy  with  the  cleanliness 

1 

2 

3 

4 

5 

2.11)  I  am  happy  with  the  quality  of  roads 

1 

2 

3 

4 

5 

2. 12)  The  government  has  met  my  expectations  for  education 

1 

2 

3 

4 

5 

2.13)  The  government  has  met  my  expectations  for  healthcare 

1 

2 

3 

4 

5 

2.14)  The  government  has  met  my  expectations  for  other  public 
services 

1 

2 

3 

4 

5 

2.15)  I  am  satisfied  with  the  public  services  in  my  community 

1 

2 

3 

4 

5 

2.16)  I  believe  my  government  is  fair 

1 

2 

3 

4 

5 

2.17)  I  am  satisfied  with  the  benefits  I  have  received  from  the 
government 

1 

2 

3 

4 

5 

Please  turn  to  next  page. 
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Section  3: 

Based  on  your  overall  experience  in  the  last  12  months, 
please  rate  your  community  on  a  scale  of  1  to  5,  with  1 
being  strongly  disagree  and  5  being  strongly  agree,  for 
each  of  the  following  items: 

Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Strongly 

Agree 

3.1)  I  am  happy  with  the  overall  quality  of  the  school 
buildings 

1 

2 

3 

4 

5 

3.2)  I  am  happy  with  the  overall  quality  of  the  school 
teachers 

1 

2 

3 

4 

5 

3.3)  I  am  happy  with  the  overall  quality  of  the  learning 
materials  available  to  students  (examples:  textbooks, 
homework  assigmnents,  and  technology) 

2 

3 

4 

5 

3.4)  I  am  happy  with  the  ability  of  schools  to  provide 
quality  education 

1 

2 

3 

4 

5 

3.5)  I  am  happy  with  the  overall  quality  of  the  hospitals 

1 

2 

3 

4 

5 

3.6)  I  am  happy  with  the  overall  quality  of  doctors  and 
nurses 

1 

2 

3 

4 

5 

3.7)  I  am  happy  with  the  overall  quality  of  the  medical 
equipment  at  the  hospitals 

1 

2 

3 

4 

5 

3.8)  I  am  happy  with  the  ability  of  hospitals  to  provide 
quality  medical  treatment 

1 

2 

3 

4 

5 

Section  4: 

On  a  scale  from  1  to  5,  with  1  being  strongly  disagree 
and  5  being  strongly  agree,  how  would  you  rate  each  of 
the  following  statements? 

Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Strongly 

Agree 

4.1)  I  am  familiar  with  the  types  of  assistance  my  country 
receives  from  the  United  States 

1 

2 

3 

4 

5 

4.2)  United  States  assistance  helps  provide  a  better  standard 
of  living  in  my  community 

1 

2 

3 

4 

5 

4.3)  United  States  training  in  Belize  is  good  for  my  country 

1 

2 

3 

4 

5 

4.4)  My  opinion  of  the  U.S.  has  improved  due  to  the 
assistance  received 

1 

2 

3 

4 

5 

Section  5:  Demographics 

S.Age)  Age  (circle  one):  under  16  16-25  26-35  36- 

45  46+ _ 

5.  Job)  Job: 

5. Children )  Number  of  children: 


5.Education)  Highest  education  level: 
S.Location)  Location  of  survey: 
5.Gender)  Gender:  Male  Female 

S.Date)  Date  of  survey: 


Thank  you  for  your  participation! 
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Appendix  B:  Research  Approval  Documentation 


DEPARTMENT  OF  THE  AIR  FORCE 
AIR  FORCE  INSTITUTE  OF  TECHNOLOGY 
WRIGHT-PATTERSON  AIR  FORCE  BASE  OHIO 


10  Apr  2014 


MEMORANDUM  FOR  DR.  AL  THAL 

FROM:  Jeffrey  A.  Ogden.  Ph.D. 

AFIT  IRB  Research  Reviewer 
2950  Hobson  Way 

Wriglit-Patterson  AFB,  OH  45433-7765 

SUBJECT:  Approval  for  exemption  request  from  human  experimentation  requirements  (32  CFR 
219,  DoDD  3216.2  and  AFI  40-402)  for  the  Impact  of  U.S.  Investment  for  Civil  Infrastructure  in 
Developing  Countries. 

1.  Your  request  was  based  on  the  Code  of  Federal  Regulations,  title  32,  part  219,  section  101, 
paragraph  (b)  (2)  Research  activities  that  involve  the  use  of  educational  tests  (cognitive, 
diagnostic,  aptitude,  achievement),  survey  procedures,  interview  procedures,  or  observation  of 
public  behavior  unless:  (i)  Information  obtained  is  recorded  in  such  a  manner  that  human 
subjects  can  be  identified,  directly  or  through  identifiers  linked  to  the  subjects;  and  (ii)  Any 
disclosure  of  the  human  subjects’  responses  outside  the  research  could  reasonably  place  the 
subjects  at  risk  of  criminal  or  civil  liability  or  be  damaging  to  the  subjects’  financial  standing, 
employability,  or  reputation. 

2.  Your  study  qualifies  for  this  exemption  because  you  are  not  collecting  sensitive  data,  which 
could  reasonably  damage  the  subjects’  financial  standing,  employability,  or  reputation.  Further, 
you  are  not  collecting  any  demographic  data  which  could  realistically  be  expected  to  map  a 
given  response  to  a  specific  subject. 

3.  This  determination  pertains  only  to  the  Federal,  Department  of  Defense,  and  Air  Force 
regulations  that  govern  the  use  of  human  subjects  in  research.  Further,  if  a  subject’s  future 
response  reasonably  places  them  at  risk  of  criminal  or  civil  liability  or  is  damaging  to  their 
financial  standing,  employability,  or  reputation,  you  are  required  to  file  an  adverse  event  report 
with  this  office  immediately. 


4/10/2014 


X 


Jeffrey  A.  Ogden,  Ph.D. 


Jeffrey  A.  Ogden,  Ph.  D. 

IRB  Exempt  Determination  Official 
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DEPARTMENT  OF  THE  AIR  FORCE 

HEADQUARTERS  UNITED  STATES  AIR  FORCE  APR  2  9  2014 

WASHINGTON  DC 


MEMORANDUM  FOR  AFIT/ENV 

ATTN:  ALFRED  E.  THAL,  JR.,  PHD 


FROM:  AFMSA/SGE-C 

Research  Oversight  &  Compliance  Division 
7700  Arlington  Blvd.  Ste.  5151 
Falls  Church,  VA  22042-5151 

SUBJECT:  Human  Research  Protection  Official  (HRPO)  Review  of  FSG20140013E 


References:  (a)  32  CFR  219,  Protection  of  Human  Subjects 

(b)  10  USC  980,  Limitation  on  Use  of  Humans  as  Experimental  Subjects 

(c)  AFI 40-402,  Protection  of  Human  Subjects  in  Research 

(d)  DoDI  3216.02,  Protection  of  Human  Subjects  and  Adherence  to  Ethical 
Standards  in  DoD-Supported  Research 

In  accordance  with  Reference  (a),  101  (b)(2)  &  Reference  (d),  the  HRPO  has  reviewed  and 
concurs  with  the  Exempt  Determination  Official  of  the  following: 

FSG20140013E,  “Support  for  the  study  titled:  The  Impact  of  U.S.  Investment  for  Civil 
Infrastructure  in  Developing  Countries”. 

Please  contact  my  office  to  discuss  any  substantive  change  to  this  activity  prior  to 
implementation  to  ensure  it  does  not  impact  the  determination  herein  or  compliance  with  the 
above  references. 


JAMES  BENJACK,  Lt  Col,  USAF,  BSC 
Director,  Research  Oversight  &  Compliance  Division 
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Appendix  C:  R  Code 


library (el  07 1 ) 

survey  <-  read. csv ( " JoelR12 . csv" ,  head=TRUE) 

#  reads  .csv  file  from  directory  to  dataframe  "survey" 

R  =  10000 

#  sets  R  iterations  of  for  loop,  insensitive 
S  =  1 

#  sets  S  iterations  of  sampling,  very  sensitive 
index  =  1 : length ( survey$El ) 

#  sets  index  vector  for  mult  variable  sampling 

rEAvg  <-  numeric (R)  #  defines  bootstrap  data  vectors 

rPAvg  <-  numeric (R) 
rDAvg  <-  numeric (R) 
rSatAvg  <-  numeric (R) 

bootData  <-  data . frame (rEAvg,  rPAvg,  rDAvg,  rSatAvg) 
#defines  bootstrap  data  frame 

for  (i  in  1:R)  #  takes  R  samples  w/  replacement 

{ 

bootindex  =  sample ( index,  S,  replace=T) 
bootData$rEAvg [ i ]  =  survey$EAvg [bootindex] 
bootData$rPAvg [ i ]  =  survey$PAvg [bootindex] 
bootData$rDAvg [ i ]  =  survey$DAvg [bootindex] 
bootData$rSatAvg [ i ]  =  survey$SatAvg [bootindex] 

} 

#  record  summary  stats  to  compare  between  actual  (sumX)  and 
bootstrap  (sumrX)  data 

x  =  5 

sumVar  <-  c("Mean",  "Median",  "StdDev", 

"Skewness",  "Kurtosis") 
sumE  <-  numeric (x) 
sumrE  <-  numeric (x) 
sumP  <-  numeric (x) 
sumrP  <-  numeric (x) 
sumD  <-  numeric (x) 
sumrD  <-  numeric (x) 
sumSat  <-  numeric (x) 
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sumrSat  <-  numeric (x) 


sumData  <-  data . frame ( sumVar,  sumE,  sumrE,  sumP,  sumrP, 
sumD,  sumrD,  sumSat,  sumrSat) 

#  defines  summary  stats  data  frame 


sumData  [1,  2] 
sumData [2,  2] 
sumData  [3,  2] 
sumData  [4,  2] 
sumData [5,  2] 

sumData  [1,  4] 
sumData  [2,  4] 
sumData [3,  4] 
sumData  [4,  4] 
sumData  [5,  4] 

sumData  [1,  6] 
sumData [2,  6] 

sumData [3,  6] 
sumData  [4,  6] 

sumData [5,  6] 

sumData  [1,  8] 
sumData  [2,  8] 

sumData [3,  8] 
sumData  [4,  8] 

sumData  [5,  8] 


mean ( survey$EAvg) 
median ( survey$EAvg) 
sd ( survey$EAvg) 
skewness ( survey$EAvg) 
kurtosis ( survey$EAvg) 

mean ( survey$PAvg) 
median ( survey$PAvg) 
sd ( survey$PAvg) 
skewness ( survey$PAvg) 
kurtosis ( survey$PAvg) 

mean ( survey$DAvg) 
median ( survey$DAvg) 
sd ( survey$DAvg) 
skewness ( survey$DAvg) 
kurtosis ( survey$DAvg) 

mean ( survey$SatAvg) 
median ( survey$SatAvg) 
sd ( survey$SatAvg) 
skewness ( survey$SatAvg) 
kurtosis ( survey$SatAvg) 


#populates  summary  table  data  from  actual  data 


for 

{ 


} 


(k  in  1:4)  #  populate  summary  table  data  from  bootstrap 


sumData [ 1 
sumData [ 2 
sumData [ 3 
sumData  [  4 
sumData  [  5 


(2*k)  +1] 
(2*k)  +1] 
(2*k)  +1] 
(2*k)  +1] 
(2*k)  +1] 


mean (bootData [ , k] ) 
median (bootData [ , k] ) 
sd (bootData [ ,  k]  ) 
skewness (bootData [ ,  k]  ) 
kurtosis (bootData [ ,  k]  ) 


write .csv (bootData,  f ile="resampleR12Avg . csv" ) 
write . csv ( sumData,  f ile=" sumR12Avg . csv" ,  row.names=T) 
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Appendix  D:  Additional  Resampling  Results 


Table  10:  Round 


Data 


Original  Data,  n  =  172 

Resampled  Data,  n  =  1C 

>,000 

Variable 

E 

P 

D 

Sat 

E 

P 

D 

Sat 

Mean 

4.35 

2.86 

2.75 

2.45 

4.34 

2.87 

2.76 

2.46 

Median 

4.42 

2.85 

2.67 

2.33 

4.33 

2.88 

2.67 

2.33 

Std  Dev 

0.68 

0.70 

0.89 

0.91 

0.68 

0.69 

0.88 

0.91 

Table  11:  Round  2  Data 


Original  Data,  n  =  455 

Resampled  Data,  n  =  1C 

>,000 

Variable 

E 

P 

D 

Sat 

E 

P 

D 

Sat 

Mean 

4.56 

2.99 

2.91 

2.62 

4.56 

2.99 

2.91 

2.63 

Median 

4.67 

3.00 

3.00 

2.67 

4.67 

3.00 

3.00 

2.67 

Std  Dev 

0.59 

0.76 

0.94 

0.97 

0.59 

0.76 

0.94 

0.97 

Table  12:  Non-Project  Location  Data 


Original  Data,  n  =  322 

Resampled  Data,  n  =  1C 

>,000 

Variable 

E 

P 

D 

Sat 

E 

P 

D 

Sat 

Mean 

4.49 

2.98 

2.85 

2.56 

4.50 

2.99 

2.87 

2.58 

Median 

4.67 

2.88 

2.67 

2.33 

4.67 

2.88 

2.67 

2.33 

Std  Dev 

0.64 

0.79 

0.95 

1.01 

0.62 

0.79 

0.95 

1.01 

Table  13:  Project  Location  Data 


Original  Data,  n  =  3C 

>2 

Resampled  Data,  n  =  1C 

>,000 

Variable 

E 

P 

D 

Sat 

E 

P 

D 

Sat 

Mean 

4.51 

2.93 

2.89 

2.60 

4.52 

2.93 

2.88 

2.60 

Median 

4.67 

2.88 

3.00 

2.33 

4.67 

2.88 

3.00 

2.33 

Std  Dev 

0.61 

0.69 

0.90 

0.89 

0.60 

0.70 

0.90 

0.89 
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Table  14:  Round  1  Data  at  a  Non-Project  Location 


Original  Data,  n  =  11 

14 

Resampled  Data,  n  =  1C 

>,000 

Variable 

E 

P 

D 

Sat 

E 

P 

D 

Sat 

Mean 

4.31 

2.71 

2.54 

2.23 

4.31 

2.71 

2.53 

2.24 

Median 

4.42 

2.67 

2.50 

2.00 

4.50 

2.63 

2.33 

2.00 

Std  Dev 

0.74 

0.68 

0.86 

0.83 

0.73 

0.67 

0.85 

0.82 

Table  15:  Round  1  Data  at  a  Project  Location 


C 

)riginal  Data,  n  =  56 

Resampled  Data,  n  =  1C 

>,000 

Variable 

E 

P 

D 

Sat 

E 

P 

D 

Sat 

Mean 

4.42 

3.13 

3.18 

2.89 

4.43 

3.13 

3.18 

2.90 

Median 

4.50 

3.00 

3.00 

3.00 

4.67 

3.00 

3.00 

3.00 

Std  Dev 

0.53 

0.66 

0.80 

0.92 

0.52 

0.65 

0.79 

0.91 

Table  16:  Round  2  Data  at  a  Non-Project  Location 


Original  Data,  n  =  2C 

>8 

Resampled  Data,  n  =  1C 

>,000 

Variable 

E 

P 

D 

Sat 

E 

P 

D 

Sat 

Mean 

4.60 

3.13 

3.02 

2.74 

4.60 

3.13 

3.03 

2.74 

Median 

4.67 

3.00 

3.00 

2.67 

4.67 

3.00 

3.00 

2.67 

Std  Dev 

0.55 

0.82 

0.96 

1.06 

0.54 

0.81 

0.96 

1.05 

Table  17:  Round  2  Data  at  a  Project  Location 


Original  Data,  n  =  247 

Resampled  Data,  n  =  1C 

>,000 

Variable 

E 

P 

D 

Sat 

E 

P 

D 

Sat 

Mean 

4.53 

2.88 

2.81 

2.52 

4.53 

2.88 

2.81 

2.53 

Median 

4.67 

2.88 

3.00 

2.33 

4.67 

2.88 

3.00 

2.33 

Std  Dev 

0.62 

0.69 

0.92 

0.88 

0.63 

0.69 

0.91 

0.88 
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